LO 46.1: Explain the process of model validation and describe best practices for the roles of internal organizational units in the validation process.
According to the Basel Committee ( 2 0 0 4 ) a rating system (or a rating model) comprises all of the methods, processes, controls, and data collection and IT systems that support the assessment of credit risk, the assignment of internal risk ratings, and the quantification of default and loss estimates.
To validate a rating model, a financial institution must confirm the reliability of the results produced by the model and that the model still meets the financial institutions operating needs and any regulatory requirements. The tools and approaches to validation are regularly reassessed and revised to stay current with the changing market and operating environment. The breadth and depth of validation should be consistent with the type of credit portfolios analyzed, the complexity of the financial institution, and the level of market volatility.
The rating model validation process includes a series of formal activities and tools to determine the accuracy of the estimates for the key risk components as well as the models predictive power. The overall validation process can be divided between quantitative and qualitative validation. Quantitative validation includes comparing ex post results of risk measures to ex ante estimates, parameter calibrations, benchmarking, and stress tests. Qualitative validation focuses on non-numerical issues pertaining to model development such as logic, methodology, controls, documentation, and information technology.
The rating model validation process requires confirmation and method of use within the financial institution. Results must be sufficiently detailed in terms of weaknesses and limitations, in the form of reports that are forwarded regularly to the internal control group
1. Basel Committee on Banking Supervision (2004 and 2006), International Convergence of
Capital Measurement and Capital Standards, A Revised Framework, Basel, Switzerland.
Page 104
2018 Kaplan, Inc.
Topic 46 Cross Reference to GARP Assigned Reading – De Laurentis, Maino, and Molteni, Chapter 5
and to regulatory agencies. At the same time, there must be a review of any anticipated remedies should the model prove to be weak. A summary of the overall process of model validation is provided in Figure 1.
Figure 1: Model Validation Process
Source: Figure 5.1. Fundamental steps in rating systems validation process. Reprinted from Developing, Validating and Using Internal Ratings, by Giacomo De Laurentis, Renato Maino, Luca Molteni, (Hoboken, New Jersey: John Wiley & Sons, 2010), p 239.
Best Practices
The Basel Committee (2004) outlined the following two key requirements regarding corporate governance and oversight:
All material aspects of the rating and estimation processes must be approved by the banks board of directors or a designated committee thereof and senior management. Those parties must possess a general understanding of the banks risk rating system and detailed comprehension of its associated management reports. Senior management must provide notice to the board of directors or a designed committee thereof of material changes or exceptions from established policies that will materially impact the operations of the banks rating system.
Senior management must also have a good understanding of the rating systems design and operation, and must approve material differences between established procedure and actual practice. Management must also ensure, on an ongoing basis, that the rating system is operating properly. Management and staff in the credit control function must meet regularly to discuss the performance of the rating process, areas needing improvement, and the status of efforts to improve previously identified deficiencies.
2018 Kaplan, Inc.
Page 105
Topic 46 Cross Reference to GARP Assigned Reading – De Laurentis, Maino, and Molteni, Chapter 5
In response to these two requirements, best practices for the roles of internal organizational units in the validation process include: 1. Senior management needs to examine the recommendations that arise from the
validation process together with analyzing the reports that are prepared by the internal audit group.
2. Smaller financial institutions require, at a minimum, a manager who is appointed to
direct and oversee the validation process.
3. The validation group must be independent from the groups that are developing and
maintaining validation models and the group(s) dealing with credit risk. The validation group should also be independent of the lending group and the rating assignment group. Ultimately, the validation group should not report to any of those groups.
4. Should it not be feasible for the validation group to be independent from designing and
developing rating systems, then the internal audit group should be involved to ensure that the validation group is executing its duties with independence. In such a case, the validation group must be independent of the internal audit group.
3.
6.
In general, all staff involved in the validation process must have sufficient training to perform their duties properly.
Internal ratings must be discussed when management reports to or meets with the credit control group.
7. The internal audit group must examine the independence of the validation group and
ensure that the validation group staff is sufficiently qualified.
8. Given that validation is mainly done using documentation received by groups dealing
with model development and implementation, the quality of the documentation is important. Controls must be in place to ensure that there is sufficient breadth, transparency, and depth in the documentation provided.
A summary of the validation and control processes involving various internal organizational units is provided in Figure 2.
Page 106
2018 Kaplan, Inc.
Topic 46 Cross Reference to GARP Assigned Reading – De Laurentis, Maino, and Molteni, Chapter 5
Figure 2: Validation and Control Processes
Models
Procedures
Tools
Management
decision
Task: model development and backtesting Owner: credit risk models
Task: credit risk procedures maintenance Owner: lending units/internal control units Task: continuous test of models/
development unit
processes/tools performance
Owner: lending unit/internal audit Organization/
Risk management/
CRO
COO
Task: operations
maintenance Owner: lending
units/IT/ internal audit
Task: lending
policy applications
Owner: central and decentralized
units/internal control units
Task: lending policy suitability
Owner: validation unit/internal audit
Lending unit/ CLO/COO
Lending unit/ CLO/CRO
Top management/Surveillance Board/Board of Directors
Basic controls
Second controls
layer
Third controls
layer
Accountability for supervisory
purposes
Source: Table 5.1. Processes and roles o f validation and control o f internal rating system. Reprinted from Developing, Validating and Using Internal Ratings, by Giacomo De Laurentis, Renato Maino, Luca Molteni, (Hoboken, New Jersey: John Wiley & Sons, 2010), p. 241.
C o m p a r i s o n o f Q u a l
i t a t i v e a n d Q u a n t i t a t i v e V a l
i d a t i o n P r o c e s s e s
Ken’s Python in Finance | Quantitative Analysis | Risk Management | Project Management | Photographer | Designer