In the following, the results of the five implemented simulation experiments are presented. The results are presented separately for all simulations. The last part of the section discusses the effect of the data sizes on the adjustment.
First, the effect of the main dataset size are compared and then the properties of two different sizes of the validation datasets (simulation 1a vs. simulation 1b and simulation 3a vs. simulation 3b) are briefly examined.
Design 1a
In simulation design 1a the size of main dataset was 500 and the size of the validation dataset was 250. Simulation 1a is based on model structures defined in Section 6.1. The bias and accuracy figures are plotted for all six models and for all three methods in Appendix Figure A.2 and numerical results are in Appendix Table A.1.
The ARB% varied between 0.18% and 7.37% for the primary interest pa-rameter β1. Models 2 and 5 have the largest ARB% values for all methods except the maximum likelihood for model 2. Recall that in models 2 and 5 the correlation between x1 and x2 was ρ12 = 0.8; other models hadρ12 = 0.2.
The RRMSE% varied between 15.30% and 30.42%. According to Appendix Figure A.2 all the methods had quite a similar accuracy pattern. Multiple imputation had the largest RRMSE% values and regression calibration had the smallest RRMSE% values.
Table 6.3 contains the ratio of RRMSE% values, which is the ratio of the RRMSE% of MI or RC to the RRMSE% of ML for primary interest parameter β1. For all models, the ratio of RRMSE% of MI was greater than 1, which means that the ML approach performed slightly better than the MI approach.
RC performed better than ML for all models. When compared to the results in the literature (Messer and Natarajan, 2008), the results confirm similar behaviour for multiple imputation when the measurement error is small. When the measurement error is larger, the results are slightly unequal compared to Messer and Natarajan (2008). In this simulation experiment, regression calibration performed better compared to Messer and Natarajan (2008): in all cases RC performed better than ML.
For other parameters, ARB% and RRMSE% values are shown in the Ap-pendix Figure A.2. Some quite large ARB% values were found for the re-gression calibration approach. Maximum likelihood performed well. The RRMSE% values were quite similar for all methods except parameterβ0, model 5 for regression calibration andβ2, model 5 for multiple imputation.
Table 6.3: Ratio of relative root mean-squared error percentages (RRMSE%) for β1 for models 1–6 in design 1a. σX|x is the SD of the measurement error, ρ12 is the correlation between x1 and x2 and ρ1Z is the correlation between x1 and covariate Z.
Ratio of RRMSE%,β1 Model σX|x ρ12 ρ1Z MI RC
1 1 0.2 0.8 1.07 0.89
2 1 0.8 0.2 1.07 0.73
3 1 0.2 0.2 1.05 0.90
4 3 0.2 0.8 1.07 0.98
5 3 0.8 0.2 1.20 0.96
6 3 0.2 0.2 1.12 0.97
β0 =β1 = 1,β2 =β3 = 0.5
All methods and all estimators had low standard errors; however, the stan-dard errors of RC estimates were almost always smaller than for MI and ML estimates.
Design 1b
In simulation design 1b, the size of the main dataset was 500 as in previous de-sign 1a but the size of the validation dataset was smaller being 50. A summary of the numerical results is presented in Appendix Table A.2.
In Appendix Figure A.3, the ARB% and RRMSE% values are plotted for all six models and for all three methods. For the primary interest parameter β1, the ARB% varied between 0.82% and 9.13%. It is interesting that multiple imputation shows the smallest and the highest ARB% values.
The RRMSE% varied between 21.70% and 72.04%. The largest RRMSE%
values are for the multiple imputation method, and the smallest are for regres-sion calibration, except for model 6, where the maximum likelihood is slightly better than RC.
For other parameters (β0, β2, β3) ARB% and RRMSE% values are also shown in Appendix Figure A.3. Regression calibration had relatively large and differing values. For almost all parameters, multiple imputation had the highest RRMSE% values.
In Table 6.4 the ratio of RRMSE% values, which is the ratio of the RRMSE%
of MI or RC to the RRMSE% of ML for primary interest parameter β1, are presented. For all models, the ratio of RRMSE% of the MI approach was greater than 1 which means that the ML approach performed slightly better than the MI approach. Compared to simulation design 1a (Table 6.3), the ratios of RRMSE% values in simulation design 1b are considerably greater.
For models 1–5, the RC approach performed better than ML.
Table 6.4: Ratio of relative root mean-squared error percentages (RRMSE%) for β1 for models 1–6 in design 1b. σX|x is the SD of the measurement error,ρ12 is the correlation between x1 and x2 and ρ1Z is the correlation between x1 and covariate Z.
Ratio of RRMSE%,β1 Model σX|x ρ12 ρ1Z MI RC
1 1 0.2 0.8 1.39 0.75
2 1 0.8 0.2 1.34 0.43
3 1 0.2 0.2 1.52 0.79
4 3 0.2 0.8 1.12 0.87
5 3 0.8 0.2 1.30 0.78
6 3 0.2 0.2 1.34 1.11
β0 =β1 = 1,β2 =β3 = 0.5
Design 2
This simulation represents a small-sample case. The size of the main dataset was 100 and the size of the validation dataset was 50. A summary of the numerical results is presented in Appendix Table A.3 and in Appendix Fig-ure A.4, the ARB% and RRMSE% values are plotted for all six models and for all three methods.
For the primary interest parameter β1, the ARB% varied between 0.64%
and 22.46%. For all six models, the regression calibration method had the smallest ARB% values. The RRMSE% for parameter β1 ranged from 40.48%
to 103.95%. The RRMSE% for the maximum likelihood method, Model 3 has an exceptionally large value.
The ratio of RRMSE% values in Table 6.5 for the MI approach were less than 1 for models 2–6, which means that MI performed better slightly better then the ML approach. For all six models, the RC approach was better than the ML approach.
For other beta parameters, regression calibration had more variation in ARB% values than the maximum likelihood and multiple imputation methods depending on the model and the parameter. The ML and MI methods had quite similar ARB% values.
Table 6.5: Ratio of relative root mean-squared error percentages (RRMSE%) for β1 for models 1–6 in design 2. σX|x is the SD of the measurement error, ρ12 is the correlation between x1 and x2 and ρ1Z is the correlation between x1 and covariate Z.
Ratio of RRMSE%,β1 Model σX|x ρ12 ρ1Z MI RC
1 1 0.2 0.8 1.01 0.83
2 1 0.8 0.2 0.93 0.58
3 1 0.2 0.2 0.53 0.39
4 3 0.2 0.8 0.95 0.86
5 3 0.8 0.2 0.87 0.62
6 3 0.2 0.2 0.86 0.67
β0 =β1 = 1,β2 =β3 = 0.5
Design 3a
In simulation design 3a, the size of the main dataset was 2,000 and the size of the validation dataset was 1,000. This simulation represents a “large sample”
case for both main and validation data. A summary of the numerical results is presented in Appendix Table A.4.
In Appendix Figure A.5, the ARB% and RRMSE% values are plotted for all six models and for all three methods. For the primary interest parameter β1, the ARB% varied between 0.24% and 8.36%. In the figure, models 2 and 5 have the largest ARB% values. The ARB% values for the multiple imputation method were stable.
The RRMSE% varied between 7.75% and 15.13%. According to Appendix figure A.5, all the methods had quite a similar pattern. Multiple imputation had the largest RRMSE% values except for model 6, and regression calibration had the smallest RRMSE% values.
For models 1–5, the ratio of RRMSE% for the MI approach was greater than 1 (Table 6.6). Thus, the ML approach performed slightly better than the MI approach for these models. According to the ratio of RRMSE%, RC was better than the ML approach for all six models.
The ARB% of other beta parameters for regression calibration varied the most, whereas the RRMSE% values had a relatively similar pattern for all methods. In regression calibration, the RRMSE% for parameter β0 was an exception.
Table 6.6: Ratio of relative root mean-squared error percentages (RRMSE%) for β1 for models 1–6 in design 3a. σX|x is the SD of the measurement error, ρ12 is the correlation between x1 and x2 and ρ1Z is the correlation between x1 and covariate Z.
Ratio of RRMSE%,β1 Model σX|x ρ12 ρ1Z MI RC
1 1 0.2 0.8 1.08 0.94
2 1 0.8 0.2 1.06 0.90
3 1 0.2 0.2 1.07 0.99
4 3 0.2 0.8 1.06 0.99
5 3 0.8 0.2 1.03 0.97
6 3 0.2 0.2 0.94 0.91
β0 =β1 = 1,β2 =β3 = 0.5
Design 3b
In simulation design 3b, the size of the main dataset was 2,000 and the size of the validation dataset was 100. A summary of the numerical results is pre-sented in Appendix Table A.5. The ARB% and RRMSE% values are plotted for all six models and for all three methods in Appendix Figure A.6.
For the primary interest parameter β1, the ARB% varied between 0.24%
and 8.36%. Models 2 and 5 have the largest ARB% values. ARB% values for the multiple imputation method were stable.
The RRMSE% varied between 7.75% and 15.13%. According to Appendix figure A.6, all the methods had quite a similar pattern. Multiple imputation had the largest RRMSE% values, except for model 6, and regression calibration had the smallest RRMSE% values.
For multiple imputation, the ratio of RRMSE% values are for all models greater than 1 (Table 6.7), thus maximum likelihood performed better than multiple imputation for primary interest parameter β1. Compared to simula-tion design 3a (Table 6.6), the ratios of RRMSE% values in simulasimula-tion design 3b are considerably greater.
For other beta parameters, the ARB% for regression calibration varied the most, whereas the RRMSE% values had a relatively similar pattern for all methods.
Table 6.7: Ratio of relative root mean-squared error percentages (RRMSE%) for β1 for models 1–6 in design 3b. σX|x is the SD of the measurement error,ρ12 is the correlation between x1 and x2 and ρ1Z is the correlation between x1 and covariate Z.
Ratio of RRMSE%,β1 Model σX|x ρ12 ρ1Z MI RC
1 1 0.2 0.8 1.67 0.81
2 1 0.8 0.2 1.56 0.48
3 1 0.2 0.2 1.84 0.90
4 3 0.2 0.8 1.25 0.94
5 3 0.8 0.2 1.52 0.81
6 3 0.2 0.2 1.54 1.24
β0 =β1 = 1,β2 =β3 = 0.5