VASP model validation

In the wake of high-profile bank collapses and increased regulatory scrutiny, banks are more reluctant to open their doors to virtual asset service providers (“VASPs”). For risk wary banks, VASPs with strong financial crime compliance (“FCC”) programs are among the most appealing customers in the cryptocurrency industry. But it can be tricky for VASPs to determine what a sound FCC program should look like, particularly in areas such as model validation, where regulatory requirements are not always clear and the considerations for transactions posted on the blockchain do not fit neatly into existing frameworks. In this article, the experts at Exiger assess and explain regulatory requirements for a model validation from across the globe, the nuances of model validations for VASPs, and why independent model validations for all VASPs should be a key component of an FCC program.

What is a Model Validation?

Automated transaction monitoring systems alerting for potential money laundering and sanctions screening systems are only useful if they operate properly. Model validation verifies that the models used to combat financial crime operate as expected, according to the business uses and objectives for which they were designed; how they are governed; their integration with the risk assessment of the institution; operational and conceptual soundness; and data quality.

According to the Federal Reserve’s SR 11-7: Guidance on Model Risk Management, model validation includes:

• Evaluation of Conceptual Soundness. This element involves assessing the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model. This step in validation should ensure that judgment exercised in model design and construction is well informed, carefully considered, and consistent with published research and with sound industry practice.
• Ongoing Monitoring. This step in validation is done to confirm that the model is appropriately implemented and is being used and performing as intended. It is essential to evaluate whether changes in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model and to verify that any extension of the model beyond its original scope is valid. Benchmarking can be used in this step to compare a given model’s inputs and outputs to estimates from alternatives.
• Outcomes Analysis. This step involves comparing model outputs to corresponding actual outcomes. Back-testing is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.

Although this guidance was directed at banks and issued long before VASPs were on regulators’ radars, these general elements of sound model validation are applicable to any financial industry.

Further, New York Department of Financial Services (“NY DFS”) Part 504 applies to VASPs registered in New York state, and even if it may not strictly apply to VASPs, it provides strong guidance on the areas of testing that should be performed. NY DFS Part 504 includes requirements that transaction monitoring and sanctions screening programs:

• Validate the integrity, accuracy, and quality of data;
• Be based on the institution’s risk assessment;
• Maintain detection scenarios that are designed to detect money laundering or other suspicious or illegal activities; and
• Perform end-to-end pre- and post-implementation testing of the systems.

The NY DFS Part 504 model validation framework may eventually be mirrored by other regulators and includes best practice requirements in model validations. However, as discussed in the next section, VASPs are not identical to banks or other financial institutions, and there are nuances to VASP model validation.

How are Model Validation Considerations for VASPs Different than for Banks?

VASPs have unique FCC considerations, and therefore they cannot be pigeon-holed into behaving exactly like banks when it comes to model validation. Everything moves with greater speed in the world of virtual assets. Information associated with virtual asset transactions can be both more and less transparent than traditional bank transactions. Blockchains provide a public, permanent and immutable ledger of transaction history. However, some coins and services, such as privacy coins and tumblers/mixers, mask the underlying person or entity behind the transaction. Further, some VASPs have lax know your customer (“KYC”) controls and may not know if malign users have accessed their platform, and some transactions involve transfers to wallets that are not hosted on an exchange at all, and thus do not have any KYC requirements.

Traditional transaction monitoring controls only generate alerts based on information accessible to the bank – i.e., transactions where the bank is a participant. Digital ledgers provide full visibility into all the transactions (and wallets) occurring for the entire history of the digital currency. This creates new opportunities and challenges. For example, you can see if the bitcoin or wallet in a specific transactions has ever been associated with a wallet with a nexus to a sanctioned entity, but how many steps away should a VASP be from that transaction so that the VASP feels comfortable stating is has no association with that sanctioned entity? The answer lies in VASP’s risk and reputational tolerance, and how transaction monitoring and sanctions screening systems are configured.

Digital ledgers offer a world of information that may inform customer risk rating models, such as the types of products customers are using or questionable transactions they may have been involved with, which are key to providing risk-based coverage. But VASPs may also encounter limitations as to what they can glean about customers because of privacy coins and tumblers, along with the nature of the issuer and the customer’s intended use of the virtual assets.

Many VASPs have experienced rapid growth in an uncertain regulatory environment and must adapt quickly to maintain compliance with regulators. This alone makes them different from most banks, where the pace of change is much slower. Many banks perform model validation on their FCC models annually or even less frequently, however with a rapid pace of change at VASPs, partial validation of those changes should be performed as needed to ensure that risk coverage is maintained. Once the offerings are more stable, validation frequency can be reduced.

Most virtual asset transaction monitoring controls are (i) software as a service (“SaaS”) blockchain monitoring models that take digital wallets as input and return alerts against various typologies, (ii) “traditional” SaaS models that monitor fiat onramps into the digital asset ecosystem, and (iii) custom-built typologies that monitor both blockchain and fiat data such as velocity. Even with such models acting as controls against money laundering risk, the VASP is still ultimately responsible for ensuring the models are adequate to the VASP’s risk assessment, especially if the business experiences rapid changes. Further, the VASP is ultimately responsible for ensuring the TM controls work as expected, even if the monitoring takes the form of a SaaS platform. This means that the regulator will not accept deficiencies in the vendor provided tool as an excuse for risk coverage gaps. VASPs can get a level of comfort with their vendors by requesting that the vendor provide them with assurance of the testing performed on their platform. Vendors may have independent model validations performed on their own systems, and they can provide the validation report as evidence that their tools are working as expected. However, while such a vendor provided assurance can support a VASP’s model validation, it is no substitution for additional testing tailored to the risks of the individual VASP.

Why Perform Model Validation as a VASP?

Virtual assets routinely make headlines as the payment of choice for bad actors engaged in scams, laundering proceeds of drug trafficking, and proliferation finance. The need for strong FCC among VASPs is now fully recognized. It is also recognized that VASPs are responsible for ensuring their transaction monitoring and sanctions screening systems are adequate, which can be accomplished through model validations. Failure to do so creates reputational harm and financial penalties. In 2022, for example, the New York Department of Financial Services levied a $30 million fine against a VASP because its “BSA/AML compliance program, including its transaction monitoring system, has significant deficiencies.”[2] This VASP did not have adequate staffing; did not transition from a manual transaction monitoring system that was adequate to its size, customers, and transaction volume; and did not devote sufficient resources to address risks specific to the company.

Further, VASPs that perform model validations for transaction monitoring and sanctions screening systems are more appealing to banks, who are aware that regulators will focus on high-risk customers during any regulatory exam, and banks must be prepared to answer questions about FCC risks with confidence. Thus, in addressing and reducing their FCC risks with model validations, VASPs reduce the FCC risks for the banks in which they keep their bank accounts, and improve their likelihood of partnering with a bank, or being onboarded as a bank’s customer.

Meet Regulatory Requirements

United States

The U.S. government has been a leader in applying its anti-money laundering and countering the financing of terror (“AML/CFT”) framework to virtual assets. U.S. regulators, including FinCEN, SEC, CFTC, and OFAC, have issued interpretive and clarifying guidance since 2013 to help financial institutions understand their compliance obligations. The last several years have seen U.S. regulators “examining financial institutions providing virtual assets-related services for compliance with registration, AML/CFT, and sanctions obligations and have taken enforcement actions against non-compliant institutions.” Such institutions have recently seen monetary penalties, enforcement actions, and settlement agreements. Because of this increased scrutiny, the U.S. Department of the Treasury has created several priority actions related to digital assets, including improving global AML/CFT regulation and enforcement, updating BSA regulations, strengthening U.S. AML/CFT supervision of virtual asset activities, holding illicit actors accountable, and engaging with the private sector.

In accordance with U.S. Department of the Treasury requirements, VASPs are subject to the Bank Secrecy Act.

VASPs doing business wholly or in substantial part in the United States qualify as money transmitters, which means they are required to comply with the BSA obligations that apply to MSBs, including registering with FinCEN; developing, implementing, and maintaining an effective AML program; filing SARs and currency transaction reports (CTRs); appointing a chief compliance officer; conducting training; and maintaining certain records. When operators of these VASPs violate the BSA or neglect regulatory requirements, such as failing to establish effective AML programs or report suspicious activities, their actions present a vulnerability to the financial system.
U.S. Department of the Treasury (2023), Digital Assets Action Plan

Increased penalties and scrutiny further highlight the need for VASPs to have transaction monitoring and sanctions screening systems in place. These will help mitigate the risk of money laundering and sanctions, and a model validation is a key part of this process. An independent model validation will bring industry experts who can benchmark model effectiveness.

At the state level, VASPs subject to NY DFS regulations must adhere to Part 504 requirements on transaction monitoring and filtering programs. These requirements include matching BSA/AML risks to the institution’s products, services, and customers; end-to-end pre- and post-implementation testing of the systems; validation of data and data sources; and governance and management oversight. Therefore, all NY DFS regulated VASPs should include model validation as part of their compliance program.

VASPs in the United States are subject to U.S. Department of the Treasury’s Financial Crimes Enforcement Network (FinCEN) and Office of Foreign Assets Control (OFAC) regulations. Cryptocurrency exchanges have faced fines, settlements, and consent orders related to violations of the Bank Secrecy Act (BSA) and other FinCEN anti-money laundering regulations.

In July 2020, the Office of the Comptroller of the Currency (OCC) issued an Interpretive Letter concluding that national banks may provide cryptocurrency custody services on behalf of customers provided it can demonstrate that it has controls in place to conduct the activity in a safe and sound manner. Specifically, the OCC supervisory office will review the bank’s risk management systems and controls and risk measurement systems. This includes having systems in place to identify, measure, monitor, and control operational and compliance risk (specifically, BSA, anti-money laundering, and sanctions requirements).

FATF

The Financial Action Task Force (“FATF”) recommends that VASPs be regulated for AML/CFT purposes, licensed or registered, and subject to effective systems for monitoring or supervision. Therefore, VASPs’ obligations match those of traditional financial institutions and include transaction monitoring and sanctions compliance.

There has been recent progress in introducing FATF’s Travel Rule, which requires the private sector to obtain/exchange beneficiary and originator information with virtual asset transfers. The Travel Rule is a key requirement that enables the private sector to comply with transaction monitoring and sanctions screening requirements. FATF considers it a key priority for the private sector to strengthen efforts to facilitate interoperability across Travel Rule technological solutions

United Kingdom

Cryptoasset businesses in the United Kingdom must be registered with the FCA. The FCA’s Money Laundering Regulations (“MLRs”) include the requirement of businesses to show that it has policies, controls, and procedures in place to effectively manage money laundering and terrorist financing risks proportionate to the size and nature of the business’s services. This includes customer due diligence, ongoing monitoring of customers and transactions, and establishing an independent internal audit function.

Singapore

The Monetary Authority of Singapore (MAS) implemented the Payment Services Act in 2020 whereby MAS issues licenses to digital payment token services. The majority of applicants have been rejected due to stringent MAS standards. The MAS requires payment service providers to have appropriate transaction monitoring systems in place, with the degree of automation and sophistication dependent on the size and complexity of the payment service provider’s operations. This system should be independently validated and the parameters and thresholds should be periodically reviewed.

Hong Kong

The Hong Kong Monetary Authority (HKMA) is currently developing a regulatory regime on crypto assets that licenses VASPs and subjects them to comprehensive regulatory framework covering anti-money laundering and counter-terrorist financing, governance, audits, risk management, and more. This regime is targeted for implementation by 2024.

Australia

Digital currency exchanges with a geographical link to Australia must register with the Australian Transactions Reports and Analysis Centre (AUSTRAC) and meet AML/CTF compliance and reporting obligations. The Australian government is developing new regulations specific to digital currency exchanges, which it proposes to call Crypto Asset Secondary Service Providers (CASSPRs). Current regulations do not specify whether digital assets are financial products and subject to the Australian Securities and Investments Commission or a consumer product regulated under the Australian Competition and Consumer Commission. The proposed obligations for CASSPRs include regular independent audits and compliance with AML/CTF provisions.