Theoretical Quantile-Quantile
4.5 Evaluation of each method
Evaluation of the above three VaR calculation methods based on 5 models will be processed in terms of test of acceptability, variability, accuracy and measurement error.
VaR calculated using Variance-Covariance method based on EARCH model will be simplified as VCEA, Variance-Covariance method based on RiskMetrics will be VCRM, Monte Carlo simulation modified by EARCH will be MCEA, Monte Carlo simulation modified by RiskMetrics will be MCRM and historical simulation will be simplified as HS in the following contents. All the results obtained for the evaluation test are realized using Microsoft Excel.
4.5.1 Acceptability
The first of acceptability test is to calculate the maximum likelihood of the unconditional correct coverage of each model introduced by Kupiec (1995). P is the loss failure rate defined by given probability, in this paper we use p=0.05, T is the total number of evaluation sample period, that is 125 and 50 days in this paper, while N is the number of
―failure‖ that occurs for each VaR calculation model. N for all the methods are reported in the Table 4
Table 4. Number of failure.
Methods VCRM VCEA MCRM MCEA HS
N (T=125) 1 4 4 5 13
N (T=50) 1 3 2 3 9
The next step is to follow the formula from Kupiec and calculate the correct unconditional coverage LRuc of each method, after that, compare these LRuc to the interval value of a LRuc which has a chi-squared distribution with one degree of freedom ( 12,a), in this paper, a=0.05 hence the interval value for LRuc is 3.841455. If a calculated LRuc is greater then 3.841455, that will leads to a rejection of acceptance of the VaR method.
The steps of calculating correct conditional coverage LRcc is similar with LRuc, however, the LRcc calculated will be compared with the interval value of LRcc which has a chi-squared distribution with two degree of freedom ( 22,a). The interval value for LRcc that under a 0.05 of confidence level is 5.991476. If an LRcc calculated is smaller than this value, then it will lead to an acceptance of the underlying VaR method.
The results of LRuc and LRcc with 125 days and 50 days evaluation sample are presented in Table 5 and Table 6.
Table 5. Results of LRuc with 125 days and 50 days evaluation sample.
Method T=125 T=50
VCRM 7.063595 1.256379
VCEA 0.972068 0.099211
MCRM 0.972086 0.112671
MCEA 0.281676 0.099211
HS 5.932733 10.98988
Table 6. Results of LRcc with 125 days and 50 days evaluation sample.
Method T=125 T=50
VCRM 7.079921 1.256379
VCEA 1.239050 0.492088
MCRM 1.239050 0.28378
MCEA 0.702303 0.492088
HS 14.26835 18.14761
4.5.2 Variability
To implement variability test, the first procedure is to calculate the MRB of each method based on approaches introduced by Engel & Gizycki (1999), the total evaluation sample periods in this paper are 125days and 50 days. VaRt is the mean value of VaR calculated using 5 different models at horizon t, and then the difference between VaR of each model with VaRt will be divided by this mean value. Finally all these difference percentage should be summarized and divided by total sample 125 and 50 and we will get the results of MRB of each model. The mean relative bias statistic captures the degree of the average bias of the VaR of the specific model from the all-model average, hence the larger the MRB value, the more variability a VaR model will have. However, the MRB measure is in terms of the relative but not absoluteconcept. The bias of the model evaluated by MRB is not absolutely. To calculate the RMSRB of each model, which is introduced by Hendricks (1997), the process is similar; the only difference is that we use absolute difference instead of the above relative one. RMSRB is a better reflection of the bias of means of estimation towards the means of all estimation methods. Hence when analyze the results; the absolute value of MRB and RMSRB will be analyzed respectively. The results of MRB and RMSRB with both 125-day and 50-day evaluation sample are showed in table 7 and table 8.
Table 7. Results of MRB with a 125-day and 50-day evaluation sample.
Method T=125 T=50
VCRM 0.242362 0.311691
VCEA 0.200751 0.173495
MCRM 0.007265 0.060061
MCEA -0.0296 -0.05162
HS -0.42478 -0.49362 Table 8. Results of RMSRB with a 125-day and 50-day evaluation sample.
Method T=125 T=50
VCRM 0.265421 0.32882
VCEA 0.27118 0.197226
MCRM 0.080147 0.103793
MCEA 0.073096 0.091711
HS 0.434964 0.49479
Due to the relative nature of MRB, we will compare and range the absolute value of MRB and RMSRB of each method to observe the variability of the 5 tested VaR calculation methods. The results of absolute MRB and RMSRB will be showed from figure 14 and figure 17.
0 0.1 0.2 0.3 0.4 0.5
MCRM MCEA VCEA VCRM HS
Methods
A b s o lu te M R B
Figure 14. Range of absolute MRB with 125-day evaluation sample.
0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6
MCEA MCRM VCEA VCRM HS
Met hods
MRB
Figure 15. Range of absolute MRB with 50-day evaluation sample.
0 0.1 0.2 0.3 0.4 0.5
MCEA MCRM VCRM VCEA HS
Methods
R M S R B
Figure 16. Range of RMSRB with 125-day evaluation sample.
0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6
MCEA MCRM VCEA VCRM HS
Met hods
RMSRB
Figure 17. Range of RMSRB with 50-day evaluation sample.
4.5.3Accuracy
According to Binary loss function, first the forecast daily VaR with the realized return of index for a 125-day and 50-day evaluation window should be compared. If the realized daily return exceeds the value of forecast VaR, it is considered as failure or exception and gives a number of 1; a number of 0 will be given for other cases. Then summarize the number of ―failure‖ and divide this number by 125 and 50 of total sample and we will get the value from Binary loss function, which is marked as BLF in the table 9.
The key of a binary loss function is the summarize number of failure or exception, it only considers the number that an exception happens but doesn’t consider about the serious that exception happens, a equal weight is given to all the forecast VaR that is small then the realized return. If the VaR calculated using particular model truly reflects the risk situation under given probability, for example, with a 5% of confidential level, the value of BLF should be 0.05 if the model is exactly reflects the real situation. The closer this value to 0.05, the more accurate the model will be.
While a Quadratic loss function also considers about the magnitude that one ―exception‖
occurs. As with Binary loss function, first we give a number of 1 to the VaR that is smaller than the realized return and 0 to others, meanwhile the magnitude will be examined by
calculating the square of difference of a VaR and a realized return when it is an exception, then summarize 1 and this calculated square difference together. Finally compute all these numbers and divided by 125 and 50 of total sample days and we will get the result of a quadratic loss, which is expressed as QLF and showed in the three column of the following table. A VaR model that has a QLF number equals or very close to 0.05 will be a good model in this case. Results of QLF with 125-day and 50-day evaluation sample will be reported in table 10.
Table 9. Results of BLF with 125-day and 50-day evaluation sample.
Table 10. Results of QLF with 125-day and 50-day evaluation sample.
4.5.4 Measurement error test
By regressing hit of each VaR calculation method with its own lags, it is noticeable that there is no autocorrelation for all the lags of hit. While when examine the correlation between the hit and a constant, as well as the correlation between the hit and the current VaR, a QLS regression method was used. The results of Hit test with 125-day evaluation sample and 50-day evaluation sample are showed in table 11 and table 12 respectively.
Methods T=125 T=50
VCRM 0.008 0.02
VCEA 0.032 0.06
MCRM 0.032 0.04
MCEA 0.04 0.06
HS 0.104 0.18
Methods T=125 T=50
VCRM 0.008 0.02
VCEA 0.032006 0.060014
MCRM 0.032004 0.040009
MCEA 0.040016 0.060037
HS 0.104063 0.180139
Table 11. Results of Hit test with 125-day evaluation sample.
C VaR
Methods P-value P-value
VCEA 0.2071 0.1576
VCRM 0.1540 0.5834
MCEA 0.1745 0.1501
MCRM 0.3677 0.4724
HS 0.0445 0.0576
Table 12. Results of Hit test with 50-day evaluation sample.
C VaR
Methods P-value P-value
VCEA 0.1518 0.1564
VCRM 0.8893 0.9426
MCEA 0.1518 0.1564
MCRM 0.3004 0.3194
HS 0.9571 0.9073