Portfolio Backtesting
4b. Testing backtest quality
statistically: Fitting ‘period by ‘period’
[this page | pdf | references | back links]
Return to Abstract
and Contents
Next page
4.6 Commonly, we
want the model not only to fit the data in aggregate but also to fit it ‘period
by period’. By this we mean that we want exceptionally adverse outcomes to
occur apparently randomly through time rather than being strongly clumped together
into narrow time windows. The latter might imperil the solvency of a firm more
than the former, since there would be less time during such a window to
generate new profits or raise new capital needed to maintain a solvent status
or credible business model.
4.7 Campbell
(2006) explains that the problem of determining whether a ‘hit’ sequence
(i.e. for, say, VaR, an indicator of the form 
which is 1 if the
actual outcome for time period 
is worse than the 
-quantile VaR, and 0 otherwise) is acceptable involves two key properties, namely:
(a) unconditional
coverage, i.e. actual probability of occurrence when averaged through time
should match expected probability of occurrence; and
(b) independence, i.e. that any two elements of the hit sequence should be independent of each
other.
4.8 The former
can be tested for by using, for example, Kupiec’s (1995)
test statistic as described in Campbell
(2006), which involves a proportion of failures 
,
defined as follows, where there are 
observations:
where 
= observed number of
failures, 
4.9 Alternatively
it can be tested for by using a z-statistic also described in Campbell
(2006):
4.10 Campbell
(2006) also describes several ways of testing for independence, including Chrisftofferson’s
(1998) Markov test (which examines whether the likelihood of a VaR
violation at time depends on whether or
not a VaR violation occurred at time 
by building up a
contingency table). This idea could presumably be extended to correlations
between times further apart. He also describes a more recent test suggested by Christofferson
and Pelletier (2004) which uses the insight that if VaR violations are
independent of each other then the amount of time between them should also be
independent, which hristofferson and Pelletier apparently argue may be a more
powerful test than the Markov test. Campbell
(2006) also describes ways of testing for unconditional coverage and
independence simultaneously.
NAVIGATION LINKS
Contents | Prev | Next