Validation of corporate probability of default models considering alternative use cases and the quantification of model risk

Michael Jacobs Jr.; Michael Jacobs Jr.

doi:10.3934/DSFE.2022002

Data Science in Finance and Economics

2022, Volume 2, Issue 1: 17-53. doi: 10.3934/DSFE.2022002

Previous Article Next Article

Research article

Validation of corporate probability of default models considering alternative use cases and the quantification of model risk

Michael Jacobs Jr. ^,

CFA, Lead Quantitative Analytics & Modeling Expert, Head – C & I 1^st Line Model Development Validation & Quality Assurance, PNC Financial Services Group – Balance Sheet Analytics & Modeling / Model Development, 340 Madison Avenue, New York, N.Y. 10173, U.S.A

Received: 29 January 2022 Revised: 25 March 2022 Accepted: 31 March 2022 Published: 12 April 2022
JEL Codes: G21, G28, M40, E47

In this study we consider the construction of through-the-cycle ("TTC") probability-of-default ("PD") models designed for credit underwriting uses and point-in-time ("PIT") PD models suitable for early warning uses, considering which validation elements should be emphasized in each case. We build PD models using a long history of large corporate firms sourced from Moody's, with a large number of financial, equity market and macroeconomic candidate explanatory variables. We construct a Merton model-style distance-to-default ("DTD") measure and build hybrid structural and reduced-form models to compare with the financial ratio and macroeconomic variable-only models. In the hybrid models, the financial and macroeconomic explanatory variables still enter significantly and improve the predictive accuracy of the TTC models, which generally lag behind the PIT models in that performance measure. While all classes of models have high discriminatory power by most measures, on an out-of-sample basis the TTC models perform better than the PIT models. We measure the model risk attributable to various model assumptions according to the principle of relative entropy and observe that omitted variable bias with respect to the DTD risk factor, neglect of interaction effects and incorrect link function specification has the greatest, intermediate and least impacts, respectively. We conclude that care must be taken to judiciously choose how we validate TTC vs. PIT models, as criteria may be rather different and apart from standards such as discriminatory power. This study contributes to the literature by providing expert guidance to credit risk modeling, model validation and supervisory practitioners in managing model risk.
- probability of default,
- point-in-time,
- through-the-cycle,
- credit risk,
- model validation,
- credit underwriting,
- early warning systems,
- regulatory capital,
- model risk
Citation: Michael Jacobs Jr.. Validation of corporate probability of default models considering alternative use cases and the quantification of model risk[J]. Data Science in Finance and Economics, 2022, 2(1): 17-53. doi: 10.3934/DSFE.2022002

Related Papers:

Abstract

In this study we consider the construction of through-the-cycle ("TTC") probability-of-default ("PD") models designed for credit underwriting uses and point-in-time ("PIT") PD models suitable for early warning uses, considering which validation elements should be emphasized in each case. We build PD models using a long history of large corporate firms sourced from Moody's, with a large number of financial, equity market and macroeconomic candidate explanatory variables. We construct a Merton model-style distance-to-default ("DTD") measure and build hybrid structural and reduced-form models to compare with the financial ratio and macroeconomic variable-only models. In the hybrid models, the financial and macroeconomic explanatory variables still enter significantly and improve the predictive accuracy of the TTC models, which generally lag behind the PIT models in that performance measure. While all classes of models have high discriminatory power by most measures, on an out-of-sample basis the TTC models perform better than the PIT models. We measure the model risk attributable to various model assumptions according to the principle of relative entropy and observe that omitted variable bias with respect to the DTD risk factor, neglect of interaction effects and incorrect link function specification has the greatest, intermediate and least impacts, respectively. We conclude that care must be taken to judiciously choose how we validate TTC vs. PIT models, as criteria may be rather different and apart from standards such as discriminatory power. This study contributes to the literature by providing expert guidance to credit risk modeling, model validation and supervisory practitioners in managing model risk.

References

[1]	Aguais S (2008) Designing and implementing a Basel Ⅱ compliant PIT-TTC ratings framework.
[2]	Altman EI (1968) Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. J Finance 23: 589–609. https://doi.org/10.2307/2978933 doi: 10.1111/j.1540-6261.1968.tb00843.x
[3]	Altman EI, Narayanan P (1997) An international survey of business failure classification models. Financ Mark Inst Instrum 6: 1–57. https://onlinelibrary.wiley.com/doi/abs/10.1111/1468-0416.00010
[4]	Altman EI, Rijken HA (2004) How rating agencies achieve rating stability. J Bank Financ 28: 2679–2714. https://doi.org/10.1016/j.jbankfin.2004.06.006 doi: 10.1016/j.jbankfin.2004.06.006
[5]	Altman EI, Rijken HA (2006) A point-in-time perspective on through-the-cycle ratings. Financ Anal J 62: 54–70. https://doi.org/10.2469/faj.v62.n1.4058 doi: 10.2469/faj.v62.n1.4058
[6]	Amato JD, Furfine CH (2004) Are credit ratings procyclical? J Bank Financ 28: 2641–2677. https://doi.org/10.1016/j.jbankfin.2004.06.005 doi: 10.1016/j.jbankfin.2004.06.005
[7]	Cesaroni T (2015) Procyclicality of credit rating systems: how to manage it. J Econ and Bus 82: 62–83. https://doi.org/10.1016/j.jeconbus.2015.09.001 doi: 10.1016/j.jeconbus.2015.09.001
[8]	Chava S, Jarrow RA (2004) Bankruptcy prediction with industry effects. Rev Financ 8: 537–569. https://doi.org/10.1093/rof/8.4.537 doi: 10.1093/rof/8.4.537
[9]	Cheng S, Long JS (2007) Testing for ⅡA in the multinomial logit model. Sociol Methods Res 35: 583–600. https://journals.sagepub.com/doi/abs/10.1177/0049124106292361 doi: 10.1177/0049124106292361
[10]	Duffie D, Singleton K (1998) Simulating correlated defaults. Paper presented at the Bank of England Conference on Credit Risk Modeling and Regulatory Implications Working Paper, Stanford University. https://kenneths.people.stanford.edu/sites/g/files/sbiybj3396/f/duffiesingleton1999.pdf
[11]	Duffie D, Singleton KJ (1999) Modeling term structures of defaultable bonds. Rev Financ Stud 12: 687–720. https://doi.org/10.1093/rfs/12.4.687 doi: 10.1093/rfs/12.4.687
[12]	Dwyer, Douglass, Ahmet E. Kogacil, Roger M. Stein (2004) Moody's KMV RiskCalc^TM v2.1 Model. Moody's Analytics. Available from: https://www.moodys.com/sites/products/productattachments/riskcalc%202.1%20whitepaper.pdf
[13]	Fry TRL, Harris MN (1996) A Monte Carlo study of tests for the independence of irrelevant alternatives property. Transp Res Part B: Methodol 31: 19–32. https://journals.sagepub.com/doi/abs/10.1177/0049124106292361 doi: 10.1177/0049124106292361
[14]	Glasserman P, Xu X (2014) Robust risk measurement and model risk. Quant Financ 14: 29–58. https://doi.org/10.1080/14697688.2013.822989 doi: 10.1080/14697688.2013.822989
[15]	Hamilton DT, Sun Z, Ding M (2011) Through-the-cycle EDF credit measures. Moody Anal. https://ssrn.com/abstract=1921419
[16]	Hausman J, McFadden D (1984) Specification tests for the multinomial logit model. Econometrica J Econometric Soc 1219–1240. https://doi.org/10.2307/1910997 doi: 10.2307/1910997
[17]	Hansen LP, Sargent TJ (2007) Robustness. Princeton: Princeton University Press, Available from: http://www.library.fa.ru/files/Robustness.pdf
[18]	Hodrick RJ, Prescott EC (1997) Postwar US business cycles: an empirical investigation. J Money, Credit Bank 1–16. https://doi.org/10.2307/2953682 doi: 10.2307/2953682
[19]	Jacobs Jr M (2020) The accuracy of alternative supervisory methodologies for the stress testing of credit risk. Int J Financ Eng Risk Manage 3: 254–296. http://michaeljacobsjr.com/files/Jacobs_2020_AccAltSupMdlsStrTstCrRisk_IJE_RM_vol3no3_pp254-296.pdf
[20]	Jacobs Jr. M (2020) A holistic model validation framework for current expected credit loss (CECL) model development and implementation. Int J Financ Stud 8: 1–36. https://doi.org/10.3390/ijfs8020027 doi: 10.3390/ijfs8010001
[21]	Jacobs Jr. M (2022) Borrower level models for stress testing corporate probability of default and the quantification of model risk. Int J Econ Finance 14: 75–99. https://doi.org/10.5539/ijef.v14n4p75 doi: 10.5539/ijef.v14n4p75
[22]	Jacobs Jr. M, Karagozoglu AK, Sensenbrenner F (2015) Stress testing and model validation: application of the Bayesian approach to a credit risk portfolio. J Risk Model Validation 9: 41–70. https://ssrn.com/abstract=2684227
[23]	Jarrow RA, Turnbull SM (1995) Pricing derivatives on financial securities subject to credit risk. J Financ 50: 53–85. https://doi.org/10.1111/j.1540-6261.1995.tb05167.x doi: 10.1111/j.1540-6261.1995.tb05167.x
[24]	Kiff MJ, Kisser M, Schumacher ML (2013) Rating through-the-cycle: what does the concept imply for rating stability and accuracy? International Monetary Fund, 2013. https://www.imf.org/external/pubs/ft/wp/2013/wp1364.pdf
[25]	Li C, Shepherd BE (2012) A new residual for ordinal outcomes. Biometrika 99: 473–480. http://dx.doi.org/10.1093/biomet/asr073 doi: 10.1093/biomet/asr073
[26]	Liu D, Zhang H (2018) Residuals and diagnostics for ordinal regression models: a surrogate approach. J Am Stat Assoc 113: 845–854. https://doi.org/10.1080/01621459.2017.1292915 doi: 10.1080/01621459.2017.1292915
[27]	Löffler G (2004) An anatomy of rating through the cycle. J Bank Financ 28: 695–720. https://doi.org/10.1016/S0378-4266(03)00041-4 doi: 10.1016/S0378-4266(03)00041-4
[28]	Löffler G (2013) Can rating agencies look through the cycle? Rev Quant Financ Account 40: 623–646. https://doi.org/10.1007/s11156-012-0289-9 doi: 10.1007/s11156-012-0289-9
[29]	Merton RC (1974) On the pricing of corporate debt: The risk structure of interest rates. Journal Financ 29: 449–470. https://doi.org/10.2307/2978814 doi: 10.2307/2978814
[30]	Mester LJ (1997) What's the point of credit scoring? Federal Reserve Bank of Philedelphia Business Review, 3–16. https://fraser.stlouisfed.org/files/docs/historical/frbphi/businessreview/frbphil_rev_199709.pdf
[31]	Rubtsov M, Petrov A (2016) A point-in-time-through-the-cycle approach to rating assignment and probability of default calibration. J Risk Model Validation 10: 83–112. doi: 10.21314/JRMV.2016.154
[32]	Repullo R, Saurina J, Trucharte C (2010) Mitigating the Pro-cyclicality of Basel Ⅱ. Econ Policy 25: 659–702. https://doi.org/10.1111/j.1468-0327.2010.00252.x doi: 10.1111/j.1468-0327.2010.00252.x
[33]	Small K A, Hsiao C (1985) Multinomial logit specification tests[J]. Int Econ Rev 26: 619–627. https://doi.org/10.2307/2526707 doi: 10.2307/2526707
[34]	The Bank for International Settlements—Basel Committee on Banking Supervision (BIS) (2005) Studies on the Validation of Internal Rating Systems. Working Paper 14. Basel: The Bank for International Settlements—Basel Committee on Banking Supervision. Available from: https://www.bis.org/publ/bcbs_wp14.htm.
[35]	The Bank for International Settlements—Basel Committee on Banking Supervision (BIS) (2006) International Convergence of Capital Measurement and Capital Standards: A Revised Framework. Basel: The Bank for International Settlements—Basel Committee on Banking Supervision. Available from: https://www.bis.org/publ/bcbsca.htm.
[36]	The Bank for International Settlements—Basel Committee on Banking Supervision (BIS). 2011. Basel Ⅲ: A Global Regulatory Framework for More Resilient Banks and Banking Systems. Basel: The Bank for International Settlements—Basel Commit-tee on Banking Supervision. Available from: https://www.bis.org/publ/bcbs189.htm.
[37]	The Bank for International Settlements—Basel Committee on Banking Supervision (BIS) (2016) Reducing Variation in Credit Risk-Weighted Assets—Constraints on the Use of Internal Model Approaches. Consultative Document. Basel: The Bank for International Settlements—Basel Committee on Banking Supervision. Available from: http://www.bis.org/bcbs/publ/d362.htm.
[38]	The European Banking Authority (2016) The European Banking Authority. Guidelines on PD Estimation, LGD Estimation and the Treatment of Defaulted Exposures. Consultation Paper. Paris: The European Banking Authority. Available from: https://www.eba.europa.eu/regulation-and-policy/model-validation/guidelines-on-pd-lgd-estimation-and-treatment-of-defaulted-assets.
[39]	Topp R, Perl R (2010) Through-the-Cycle Ratings Versus Point-in-Time Ratings and Implications of the Mapping Between Both Rating Types. Financ Mark Inst Instrum 19: 47–61. https://doi.org/10.1111/j.1468-0416.2009.00154.x doi: 10.1111/j.1468-0416.2009.00154.x

Reader Comments

Your name:*

Email:*
© 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)