LO 3.6: Describe the Basel rules for backtesting.
In the backtesting process, we attempt to strike a balance between the probability of a Type I error (rejecting a model that is correct) and a Type II error (failing to reject a model that is incorrect). Thus, the Basel Committee is primarily concerned with identifying whether exceptions are the result of bad luck (Type I error) or a faulty model (Type II error). The Basel Committee requires that market VaR be calculated at the 99% confidence level and backtested over the past year. At the 99% confidence level, we would expect to have 2.3 exceptions (230 x 0.01) each year, given approximately 250 trading days.
Regulators do not have access to every parameter input of the model and must construct rules that are applicable across institutions. To mitigate the risk that banks willingly commit a Type II error and use a faulty model, the Basel Committee designed the Basel penalty zones presented in Figure 5. The committee established a scale of the number of exceptions and corresponding increases in the capital multiplier, k. Thus, banks are penalized for exceeding four exceptions per year. The multiplier is normally three but can be increased to as much as four, based on the accuracy of the banks VaR model. Increasing k significantly increases the amount of capital a bank must hold and lowers the banks performance measures, like return on equity.
Notice in Figure 5 that there are three zones. The green zone is an acceptable number of exceptions. The yellow zone indicates a penalty zone where the capital multiplier is increased by 0.40 to 1.00. The red zone, where 10 or more exceptions are observed, indicates the strictest penalty with an increase of 1 to the capital multiplier.
Figure 5: Basel Penalty Zones
N um ber o f Exceptions
M ultiplier (k)
Zone
Green
Yellow
0 to 4
5
6
7
8
9
3.00
3.40
3.50
3.65
3.75
3.85
4.00
2018 Kaplan, Inc.
Page 33
Red
10 or more
Topic 3 Cross Reference to GARP Assigned Reading – Jorion, Chapter 6
As shown in Figure 3, the yellow zone is quite broad (five to nine exceptions). The penalty (raising the multiplier from three to four) is automatically required for banks with 10 or more exceptions. However, the penalty for banks with five to nine exceptions is subject to supervisors discretions, based on what type of model error caused the exceptions. The Committee established four categories of causes for exceptions and guidance for supervisors for each category:
The basic integrity o f the model is lacking. Exceptions occurred because of incorrect data or errors in the model programming. The penalty should apply.
Model accuracy needs improvement. The exceptions occurred because the model does not
accurately describe risks. The penalty should apply. Intraday trading activity. The exceptions occurred due to trading activity (VaR is based on static portfolios). The penalty should be considered. Bad luck. The exceptions occurred because market conditions (volatility and correlations among financial instruments) significantly varied from an accepted norm. These exceptions should be expected to occur at least some of the time. No penalty guidance is provided.
Although the yellow zone is broad, an accurate model could produce five or more exceptions 10.8% of the time at the 99% confidence level. So even if a bank has an accurate model, it is subject to punishment 10.8% of the time (using the required 99% confidence level). However, regulators are more concerned about Type II errors, and the increased capital multiplier penalty is enforced using the 97% confidence level. At this level, inaccurate models would not be rejected 12.8% of the time (e.g., those with VaR calculated at the 97% confidence level rather than the required 99% confidence level). While this seems to be only a slight difference, using a 99% confidence level would result in a 1.24 times greater level of required capital, providing a powerful economic incentive for banks to use a lower confidence level. Exemptions may be excluded if they are the result of bad luck that follows from an unexpected change in interest rates, exchange rates, political event, or natural disaster. Bank regulators keep the description of exceptions intentionally vague to allow adjustments during major market disruptions.
Industry analysts have suggested lowering the required VaR confidence level to 93% and compensating by using a greater multiplier. This would result in a greater number of expected exceptions, and variances would be more statistically significant. The one- year exception rate at the 95% level would be 13, and with more than 17 exceptions, the probability of a Type I error would be 12.5% (close to the 10.8% previously noted), but the probability of a Type II error at this level would fall to 7.4% (compared to 12.8% at a 97.5% confidence level). Thus, inaccurate models would fail to be rejected less frequently.
Another way to make variations in the number of exceptions more significant would be to use a longer backtesting period. This approach may not be as practical because the nature of markets, portfolios, and risk changes over time.
Page 34
2018 Kaplan, Inc.
Topic 3 Cross Reference to GARP Assigned Reading – Jorion, Chapter 6
K e y C o n c e p t s
LO 3.1
Backtesting is an important part of VaR model validation. It involves comparing the number of instances where the actual loss exceeds the VaR level (called exceptions) with the number predicted by the model at the chosen level of confidence. The Basel Committee requires banks to backtest internal VaR models and penalizes banks with excessive exceptions in the form of higher capital requirements.
LO 3.2
VaR models are based on static portfolios, while actual portfolio compositions are dynamic and incorporate fees, commissions, and other profit and loss factors. This effect is minimized by backtesting with a relatively short time horizon such as daily holding periods. The backtesting period constitutes a limited sample, and a challenge for risk managers is to find an acceptable level of exceptions.
LO 3.3
The failure rate of a model backtest is the number of exceptions divided by the number of observations: N / T. The Basel Committee requires backtesting at the 99% confidence level over the past year (230 business days). At this level, we would expect 230 x 0.01, or 2.5 exceptions.
LO 3.4
In using backtesting to accept or reject a VaR model, we must balance the probabilities of two types of errors: a Type I error is rejecting an accurate model, and a Type II error is failing to reject an inaccurate model. A log-likelihood ratio is used as a test for the validity of VaR models.
LO 3.5
Unconditional coverage testing does not evaluate the timing of exceptions, while conditional coverage tests review the number and timing of exceptions for independence. Current market or trading portfolio conditions may require changes to the VaR model.
LO 3.6
The Basel Committee penalizes financial institutions when the number of exceptions exceeds four. The corresponding penalties incrementally increase the capital requirement multiplier for the financial institution from three to four as the number of exceptions increase.
2018 Kaplan, Inc.
Page 35
Topic 3 Cross Reference to GARP Assigned Reading – Jorion, Chapter 6
C o n c e p t C h e c k e r s
1.
2.
3.
4.
5.
In backtesting a value at risk (VaR) model that was constructed using a 97.3% confidence level over a 232-day period, how many exceptions are forecasted? A. 2.5. B. 3.7. C. 6.3. D. 12.6.
Unconditional testing does not reflect the: A. size of the portfolio. B. number of exceptions. C. confidence level chosen. D. timing of the exceptions.
Which of the following statements regarding verification of a VaR model by examining its failure rates is false? A. The frequency of exceptions should correspond to the confidence level used for
the model.
B. According to Kupiec (1995), we should reject the hypothesis that the model is
correct if the log-likelihood ratio (LR) > 3.84.
C. Backtesting VaR models with a higher probability of exceptions is difficult
because the number of exceptions is not high enough to provide meaningful information.
D. The range for the number of exceptions must strike a balance between the
chances of rejecting an accurate model (a Type I error) and the chances of failing to reject an inaccurate model (a Type II error).
The Basel Committee has established four categories of causes for exceptions. Which of the following does not apply to one of those categories? A. The sample is small. B. Intraday trading activity. C. Model accuracy needs improvement. D. The basic integrity of the model is lacking.
A risk manager is backtesting a sample at the 95% confidence level to see if a VaR model needs to be recalibrated. He is using 252 daily returns for the sample and discovered 17 exceptions. What is the 2;-score for this sample when conducting VaR model verification? A. 0.62. B. 1.27. C. 1.64. D 2.86.
Page 36
2018 Kaplan, Inc.
Topic 3 Cross Reference to GARP Assigned Reading – Jorion, Chapter 6
C o n c e p t C h e c k e r An s w e r s
1. C
(1 – 0.975) X 252 = 6.3
2. D Unconditional testing does not capture the timing of exceptions.
3. C Backtesting VaR models with a lower probability of exceptions is difficult because the number
of exceptions is not high enough to provide meaningful information.
4. A Causes include the following: bad luck, intraday trading activity, model accuracy needs
improvement, and the basic integrity of the model is lacking.
5. B The z-score is calculated using x = 17, p = 0.05, c = 0.95, and N = 252, as follows:
1 7 -0 .0 5 (2 5 2 ) _ 1 7 -1 2 .6 _
4.4
_ ^ ^
~ ^0.05(0.95)252 ~ V ll.9 7 ~~ 3.4598 ~~
2018 Kaplan, Inc.
Page 37
The following is a review of the Market Risk Measurement and Management principles designed to address the learning objectives set forth by GARP. This topic is also covered in:
VaR M a p p i n g
E x a m F o c u s
Topic 4
This topic introduces the concept of mapping a portfolio and shows how the risk of a complex, multi-asset portfolio can be separated into risk factors. For the exam, be able to explain the mapping process for several types of portfolios, including fixed-income portfolios and portfolios consisting of linear and nonlinear derivatives. Also, be able to describe how the mapping process simplifies risk management for large portfolios. Finally, be able to distinguish between general and specific risk factors, and understand the various inputs required for calculating undiversified and diversified value at risk (VaR).
T h e M a p p i n g P r o c e s s