Calibration (EBA/GL/2017/16)
About this app
Predicted PD vs. observed default rate, binomial confidence bands and traffic light per rating grade.
The idea
A rating model is well calibrated when its predicted probabilities of default (PD) match the default rates actually observed, per rating grade and at portfolio level. Because defaults are rare, the observed rate fluctuates randomly. So the PD is not compared with a single number; instead you check whether it lies within the confidence interval of the observation. According to the app’s info panel, the EBA guidelines (EBA/GL/2017/16, paragraphs 80–93) require a comparison with the long-run default rate and statistical tests; deviations lead to recalibration or a MoC.
What the app shows
A chart with seven rating grades: x is the predicted PD, y the observed default rate, both on a log scale. The diagonal represents perfect calibration, and each point carries a bar for the Wilson confidence interval. If the PD lies inside the interval, the grade is green; slightly outside, yellow; clearly outside, red. A panel lists PD, observed rate and status per grade, along with the number of calibrated grades and the average bias.
Try it yourself
The “model drift” slider shifts all predicted PDs by up to ±50 %, simulating systematic over- or underestimation. The sample size multiplier (25 to 400 %) shows how strongly the interval depends on N: with small samples almost every grade passes, with large ones even small deviations stand out. You can also switch the confidence level between 95 % and 99 %. The data are fictional.
Frequently asked questions
- What does calibration mean for a PD model?
- Calibration describes whether the default probabilities estimated by the model match the observed default rates on average. A model can separate good and bad borrowers well and still systematically over- or underestimate the level of PD. Discriminatory power and calibration are therefore tested separately.
- How do you test the calibration of a rating grade?
- You compare the predicted PD with the observed default rate k/N while accounting for random variation, for example with a binomial test or a confidence interval such as the Wilson interval. If the PD lies outside the interval, the deviation is statistically conspicuous. Tests across all grades, such as Hosmer–Lemeshow or Spiegelhalter, are used in addition.
- Why use the Wilson interval instead of the normal approximation?
- With small default rates and small samples, the simple normal approximation produces implausible intervals, for example ones that extend below zero. The Wilson interval stays between 0 and 1 and is considerably more reliable for rare events.
- What does the PD calibration app show?
- It compares predicted PD and observed default rate with Wilson confidence intervals for seven fictional rating grades and colours each grade as calibrated, borderline or miscalibrated. With model drift, sample size and confidence level you explore when a deviation counts as significant.
Subject: IRBA / banking supervision
More from IRBA / banking supervision
- CreditMetrics — credit portfolio model
- Data representativeness (pool ↔ bank)
- Margin of Conservatism (EBA/GL/2017/16)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps