View and download the notebook here!
How to Estimate Model Parameters with MSM
This guide contains an estimation exercise for the use of respy’s method of simulated moments interface with estimagic’s optimization capabilities.
The theoretical basis for the Method of Simulated Moments (MSM) can be found in McFadden (1989). A detailed guide to the MSM interface in respy is linked below. It provides a overview of respy’s MSM functions and outlines how inputs may be specified to construct a criterion function. This guide as a next step showcases a small estimation exercise to estimate parameters with the specified criterion function.
To how-to guide
Find out more about Method of Simulated Moments (MSM).
Motivation
Estimation of structural models is a tedious and complicated task, as it involves the optimization of a usually non-smooth criterion function with respect to a large number of parameters. Optimizers may easily get stuck in local optima for such criterion functions, preventing the finding of a global solution. It is therefore worthwhile to make sure an estimation setup is correctly specified.
The purpose of this exercise is to demonstrate how we can check in a very controlled environment whether an estimation procedure and criterion function are correctly specified for the estimation of a model. This notebook, therefore, is not a guide to estimating a model in general but instead showcases a test of the setup that should precede actual estimation. The exercise also allows us to get a sense of how sensitive the estimation process is to different calibration choices in the
optimization like bounds and algorithms, as well as the specification of the criterion function in regards to the choice of weighting matrix, the set of moments, or size of the simulated sample.
In the exercise, we will work with a simulated model which we have complete control of and for which we know the true parameter vector. We will attempt to estimate a misspecified version of the model to perturb the true parameter vector in a controlled fashion and then try to retrieve the parameters for the correct model specification using the perturbed vector as starting values. The exercise thus allows us to test whether the general setup of our optimization problem works and an optimizer is
successful in approaching the optimum for a correctly specified model (Eisenhauer et al., 2015; Eisenhauer, 2019).
Setup Estimation
Note: This guide was created using version 0.0.31 of estimagic and version 2.0.0 of respy. Please note that the guide is not updated regularly so using newer versions of the packages may require adjustments to the code.
Estimation Exercise
Finally, we can perform the estimation exercise. The target of this exercise is to test whether an optimizer manages to retrieve the true parameter vector (approximately) after we have induced it to distance itself from the true values. To generate starting values for the estimation that differ from the true vector, we misspecify the model by fixing the discount factor to 0, thus rendering agents myopic in our model. We then attempt to estimate the parameters for the misspecified model but will
fail to retrieve them as the discount factor is fixed to zero. Within the bounds chosen above, the estimated values for the free parameters will thus diverge from the true ones we started with.
In the next step, delta
is fixed back to the correct value and the resulting parameter vector from the last step is used as the starting vector for the new estimation. During this estimation, the simulated moments should converge back to the observed ones.
In short, the exercise comprises the following steps.
Begin with the true parameter vector, set delta
to 0, and fix it in the constraints, thereby misspecifying the model.
Estimate the free parameters for the misspecified model for a selected number of maximum evaluations of the criterion function.
Using the resulting parameter vector from (2) as start values, set delta
back to its true value.
Estimate the parameters using the vector from (3) as start values.
We use the function above to perform the exercise. The function returns a dictionary with the parameter vector at different steps in the exercise. Additionally, the optimizations are monitored in the databases logging_away.db
and logging_back.db
for step (2) and step (4) respectively.
Results
To evaluate the results of our exercise, there are multiple metrics we should assess. The first and easiest one is the criterion value itself. Since the objective of the optimization is to minimize the the weighted squared difference between empirical and simulated moments, the criterion value should decrease during optimization. If this is not the case, we have an immediate indicator that we made a mistake in our configuration.
Criterion Value
The criterion values during our exercise decrease for both optimizations. Interestingly, the value is actually lower at step (2) for the misspecified model that at step (4) for the correctly specified model. Only looking at the criterion value thus might lead to wrong conclusions about the parameter estimates. We therefore in the next section turn to the model moments implied by our estimates.
Criterion value for (1) Start value with delta=0: 766160.6
Criterion value for (2) Result with delta=0: 58334.3
Criterion value for (3) New starting vector correct delta: 297888.4
Criterion value for (4) Final result: 85052.4
Moments
After inspecting the criterion values, we are interested in how well the estimated parameters manage to fit the moments we selected for estimation, especially the choices of individuals in each period. We thus compute the moments for all parameter vectors that were saved to params_dict
. The function get_moment_errors_func
can generate a criterion function that returns simulated moments in addition to the function value. We thus generate a new criterion function with the argument
return_simulated_moments
set to True.
Now we can use the function to compute and plot the simulated moments at each step in the exercise.
The function defined above will generate the following subplots:
Start value with delta
= 0: The moments for the true parameter vector with the exception that delta
is set to 0.
Result with delta
= 0: The moments for the parameter vector we generate by estimating the misspecified model.
New starting vector correct delta
: The moments for the estimated parameter vector with delta
set back to its true value, i.e. the starting values for the final estimation.
Final result: The moments for the estimated parameter vector using the starting values generated in the prior step.
Observed: The “true” moments, i.e. the moments generated from the true parameter vector.
The central plot is (4) Final result since it shows the result at the end of the exercise. We want it to match the “observed” moments as closely as possible. The results overall indicate that the estimation mechanism seems to work, which is what we wanted to test. The criterion function manages to select parameters that improve the match between simulated and observed moments compared to the starting values it is faced with.
The plots also showcase some interesting aspects about the economics of the model. For example, in the first two plots where delta
is fixed to zero, no agent chooses education in any period since agents are not forward-looking for this specification.
Difference in Moments
We can also easily compute the difference in moments between the simulated and observed data. For the choices we just look at the total difference in choice frequencies for each period.
For the wage distribution we compute the relative deviation compared to the values of the empirical moments.
Again, we can see that the overall result is close to the observed data. However, for the current estimates the number of individuals choosing education in earlier periods is too low and the number of individuals choosing to work is too high. Furthermore, the mean wage is too high in every period.
Parameter History
After assessing the criterion value and moment fit of the estimates, we now turn to the parameter values themselves. We can use the parameter history that estimagic
saves in the logging databases during optimization to get a picture of how the estimates develop during the exercise and evaluate, which parameters change the most.
The plots below show the parameter history of parameters that affect wages and the non-pecuniary rewards for education and the home option. Shocks and other parameters are purposely left out for legibility. The red line indicates true parameter value and the yellow lines show the bounds set at the beginning of the exercise. The blue lines show the parameter history during optimization of the misspecified model and the orange line shows the second optimization process where we try to retrieve the
true values. The plots reveal that most of the pictured parameters did not change much during the whole exercise. The change in moments seems to mainly be driven by the wage constants of the model which increasingly move towards the bounds during the exercise.
Seeing that the wage constants are quite high compared to the true model, it is not surprising that our current specification under-predicts the choice frequencies for the education and home option and over-predicts the choice frequencies of the working alternatives. In the next step we will thus attempt to correct this problem.
Refining Estimation Results
The results from the exercise lead us to the conclusion that the overall estimation mechanism seem to work well. The optimizer manages to select parameters that improve the match between simulated and empirical moments, even after being led into the wrong direction through a misspecified model. However, our results for the parameter estimates themselves are not satisfactory yet since the model over-predicts the choice frequencies of work options and under-predicts choice frequencies of the other
alternatives.
To refine the results, we will thus run an optimization yet again, this time only on the constants of the model. This allows the optimizer to focus only on these four parameters and will prevent it form trying to improve the fit through other parameters like shocks.
For the estimation we use the parameters from step (4) of the exercise as starting values and create new constraints that fix all parameters but the constants.
The criterion value improves substantially.
Starting Criterion Value: 85052.4
Result Criterion Value: 10110.4
The moment estimates also improve compared to the prior solution.
Finally, after correcting the constants of the model, we can free up the other parameters for estimation one last time.
The criterion value improves yet again and the simulated moments seem to match the observed moments fairly well, too. We can still see some discrepancies, especially for the choice frequencies of occupation b
which could be removed through further fine-tuning. For the sake of this exercise, we are content with the result.
Starting Criterion Value: 10110.4
Result Criterion Value: 3404.1
References
Eisenhauer, P., Heckman, J. J., & Mosso, S. (2015). Estimation of dynamic discrete choice models by maximum likelihood and the simulated method of moments. International Economic Review, 56(2), 331-357.
Eisenhauer, P. (2019). The approximate solution of finite‐horizon discrete‐choice dynamic programming models. Journal of Applied Econometrics, 34(1), 149-154.
Keane, M.P. and Wolpin, K.I. (1994). The Solution and Estimation of Discrete Choice Dynamic Programming Models by Simulation and Interpolation: Monte Carlo Evidence. The Review of Economics and Statistics, 76(4): 648-672.
McFadden, D. (1989). A Method of Simulated Moments for Estimation of Discrete Response Models without Numerical Integration. Econometrica: Journal of the Econometric Society, 995-1026.
Additional Resources
Adda, J., & Cooper, R. W. (2003). Dynamic Economics: Quantitative Methods and Applications. MIT press.
Andrews, I., Gentzkow, M., & Shapiro, J. M. (2017). Measuring the Sensitivity of Parameter Estimates to Estimation Moments. The Quarterly Journal of Economics, 132(4), 1553-1592.
Davidson, R., & MacKinnon, J. G. (2004). Econometric Theory and Methods (Vol. 5). New York: Oxford University Press.
Evans, R. W. (2018, July 5). Simulated Method of Moments (SMM) Estimation. QuantEcon Notes.
Gourieroux, M., & Monfort, D. A. (1996). Simulation-based econometric methods. Oxford university press.
Stern, S. (1997). Simulation-based estimation. Journal of Economic Literature, 35(4), 2006-2039.