Model Risk, How Bad Can It Be?

How Bad Can Model Risk Be?

Have you heard how a missing hyphen resulted in a NASA spacecraft being destroyed by another rocket? Or how a NASA communication craft missed its target and crashed into Mars because units were not converted to metric. Or how Barclay’s picked up an extra 179 contracts from Lehman Brothers for nothing because of a spreadsheet copy error. These are excellent examples of model risk.

And my favorite. The London Whale fiasco. JP Morgan lost $6 billion on credit derivatives partly because a spreadsheet calculation error used the sum of two rates instead of an average.

You can find more details on each of these model risk events in Matthew Zeitlin’s article “7 Data Disasters More…” posted to Bloomberg on April 17, 2013.

These are not small numbers here. They are huge. But the errors, or mistakes, or problem with the models were small. So small they were overlooked by some of the world’s top professionals.

I had the opportunity once to review the credit risk being assigned to a new initiative. The business unit had obtained all the data and modeled out the expected losses. They ended up determining the Economic Capital necessary to underpin the initiative was around $10 million and the return on this capital was in the 30% range. Well above the target levels.

The problem was the range of data the model was using. The model was relying on 6 years of default rates. The 6 years being used were the lowest default rates in history. When this was identified and I asked the model be expanded to include at least the last credit crisis, no one liked the outcome.

What started out as a reasonable $10 million Economic Capital initiative with a 30% return, turned into $50 million Economic Capital and 6% return. Well below target levels.

At the time the business unit was not happy they could not go ahead with the initiative. However, in hindsight, having gone through the worst credit crisis on record, the business unit has saved themselves a $40 million model risk loss.

If you use spreadsheets or programs to make decisions on, ask yourself:

  • What if my model has an error in it?
  • What if the output is wrong?
  • What is the cost to the business if it is wrong?
  • Is it worth spending the time to vet the model?

It may be a missing hyphen, a calculation error or simply referring to the wrong cell. If you rely on the output to make decisions, you need to be sure there is no model risk. Otherwise you risk crashing a communication craft into the side of Mars simply because the numbers you see are in pounds when you think they are in newtons. That was a $125 million mistake.

Copyright 2012. Bruce Everitt. All rights reserved.

Bruce Everitt, a CFA® charterholder, is the Principal of Willo Consulting Group, and specializes in increasing business value for small and medium businesses around the world.

Please like, tweet or share this post if you found it interesting.


Leave a Comment

Your email address will not be published. Required fields are marked *