Documentation and Model Deficiency are Top Concerns

Entering an outsourcing contract with a model vendor means more oversight, more communication, and more risk monitoring tasks. It can also mean assuming more risk.  During RMA’s Model Risk Management Summit in December 2021, speakers for the Vendor Model Risk Management session highlighted two specific issues when working with third-party models: insufficient documentation and deficiencies found in statistical testing results conducted by vendors.

Model risk teams need to secure sufficient documentation to comply with regulatory guidelines. Model users also often require additional explanations for model development, governance, and testing. However, vendors don’t always provide detailed information.

In RMA’s recent Vendor Model Validation and Third-Party Risk Management Survey, 76% of respondents said that they found deficiencies in the statistical test results, while 72% found fault with the description of analytics, and 67% of respondents found the description of theory used in a model development to be deficient.

Source: Risk Management Association

Note: n=58

Even if a vendor agrees to provide documentation as part of their procurement contract with a bank, vendors and model users may not align on what constitutes adequate documentation. For example, a procurement team may focus on specific terms used by the regulatory bodies for model risk management. Vendors, however, may ask about terms such as “robustness” of model quality as per SR 11-7 and the Comptroller’s Handbook, and if it is adequate to include that in the contract.

Once the procurement team identifies there is a model involved in the contract, they will make sure the contract includes a clause which enables the bank to obtain documentation on model development and ongoing support. “If you don’t incorporate some of these clauses in your contract, you missed the boat,” one panelist said.

Model deficiency is a bigger issue. Risk officers often find deficiencies in statistical testing results conducted by some model vendors. For instance, if a vendor model is using faulty variables to draw a conclusion, that model the bank just purchased is no longer usable. In the case of credit risk assessment for banks, a deficient model may overlook asset-specific risks, assign inadequate risk ratings, and may not even recognize signals for credit losses.

Despite more oversight and risk management required by contracting a third-party, banks opt for external vendor models for the benefits they provide. Outsourcing helps banks to focus on core capabilities and leverage advanced modeling techniques available to them.

RMA continues to provide members resources for leading practices in model risk management, including the release of the latest survey on vendors, vendor models, and third-party risk management. We will be publishing the results of our Vendor Model Validation and Third-Party Risk Management Survey in April, 2022. Stay tuned.

Have more vendor model questions? Ask our Model Validation team.