/

### Portfolio Backtesting 4b. Testing backtest quality statistically: Fitting ‘period by ‘period’

Next page

4.6          Commonly, we want the model not only to fit the data in aggregate but also to fit it ‘period by period’. By this we mean that we want exceptionally adverse outcomes to occur apparently randomly through time rather than being strongly clumped together into narrow time windows. The latter might imperil the solvency of a firm more than the former, since there would be less time during such a window to generate new profits or raise new capital needed to maintain a solvent status or credible business model.

4.7          Campbell (2006) explains that the problem of determining whether a ‘hit’ sequence (i.e. for, say, VaR, an indicator of the form  which is 1 if the actual outcome for time period  is worse than the -quantile VaR, and 0 otherwise) is acceptable involves two key properties, namely:

(a)    unconditional coverage, i.e. actual probability of occurrence when averaged through time should match expected probability of occurrence; and

(b)   independence, i.e. that any two elements of the hit sequence should be independent of each other.

4.8          The former can be tested for by using, for example, Kupiec’s (1995) test statistic as described in Campbell (2006), which involves a proportion of failures , defined as follows, where there are  observations:

where  = observed number of failures,

4.9          Alternatively it can be tested for by using a z-statistic also described in Campbell (2006):

4.10        Campbell (2006) also describes several ways of testing for independence, including Chrisftofferson’s (1998) Markov test (which examines whether the likelihood of a VaR violation at time  depends on whether or not a VaR violation occurred at time  by building up a contingency table). This idea could presumably be extended to correlations between times further apart. He also describes a more recent test suggested by Christofferson and Pelletier (2004) which uses the insight that if VaR violations are independent of each other then the amount of time between them should also be independent, which hristofferson and Pelletier apparently argue may be a more powerful test than the Markov test. Campbell (2006) also describes ways of testing for unconditional coverage and independence simultaneously.