choice-metrics.com

Dear All,

I have designed a preliminary survey for an (mnl, d) specification. However, the final model to be estimated will be the mixed logit.
My question is whether (a) I should use the mnl design and perform a pilot survey to obtain distribution estimates which I will then input for the rppanel specification for my final survey design,
or (b) should I test this preliminary survey for efficiency under rppanel specification, and if so what do I specify as the prior distribution estimates?

Thank you for your attention & time.


All the best,

Domilé

We have found the following:
1. With data from a pilot study, it is difficult to estimate a panel mixed logit model due to insufficient data
2. Generating an efficient design for the panel mixed logit model is extremely time consuming and therefore often infeasible.

What we do is:
1. Generate an efficient or orthogonal design for the pilot study, and estimate an MNL model to obtain priors
2. Generate an efficient design for the MNL model using these priors, but TEST this design for estimating a panel mixed logit model.

Number 2 is easy in Ngene. You specify something like
;eff = M1(mnl,d)
where M1 is your MNL model, while you also specify model M2 that is a rppanel model. See the model averaging section in the Ngene manual.
Once you run the syntax, it will only optimise for MNL, but when you inspect the design, you will also get output for the rppanel model (assuming that you have set ;rdraws and ;reps), so you can test how the model will behave under the panel random parameters model.

Michiel

Thank you for your advice!
There are a few more concerns I would appreciate a helping hand with:
(a) I was wondering how critical is the S-Estimate for the pilot? You have stated in the manual that this measure is slightly unpredictable due to the random utility assumption of the underlying model.
Obviously the sample size will be small for the pilot so I am worried about: how appropriate will my coefficient estimates be for the generation of my final design? Should I not worry about this too much?
My current generated designs obtain a relatively small d-error (circa 0.0041), however the designs are also characterized by large S Estimates.
(b) Are too many interaction terms an issue? I have several dummy variables and hence was wondering whether to include interaction effects for the most influential variables for all the levels or only the limit values will suffice?

The design does in general not have an influence on the (ratio of) coefficients, but only on the standard errors. The S-estimates are only correct if the priors you put in are correct. Your priors in the pilot study will likely be quite wrong, so I would not pay too much attention to the S-estimates, unless you are convinced that the priors are close to their true values.

Including many interactions means you will have many more parameters to estimate, especially if you have interactions across dummy variables. More parameters to estimate means you will need larger designs and/or more respondents to get reliable parameter estimates.

Dear Ngene Users and Programmers

I am now at the stage of designing my final survey based on pilot survey data of 26 respondents x 10 choice situations estimated using the standard McFadden conditional multinomial logit model. My data has 3 continuous and 3 dummy variables with 4 respective levels each.
My estimation results are statistically significant - however when I include the interaction terms specified within my initial design they disturb my coefficient signs and z statistics. In fact most of the parameters become statistically insignificant once all the interactions are included. What are the implications for my final design?
Understandably the coefficients are difficult to interpret at this stage (especially without converting to odds ratios and marginal effects) - nonetheless
I was wondering how to go about deciding whether to include all the same interactions for my final design? Should I explore the different marginal effects at different levels of the variables and decide the extent of marginal effects for each interaction?

Thank you in advance for your taking the time to read my query!

Wishing everyone a beautifully productive week

Designs are optimised for a specific model, so it is a bit tricky to decide which model to put in.
You can include priors for the main effects and perhaps set the priors for the interaction effects to zero (or close to zero). This way, at least the design is mostly optimised for the main effects, but will still be able to handle the interactions effects (although there is less optimisation on these interaction effects).

Thank you for the advice.

Model consistency throughout design and estimation is crucial for the efficiency, so as advised I will include small values for my interaction priors - perhaps proportionately to pilot estimates?

I was also wondering whether to include the coefficient estimates found with or without the interactions? Logically it would not make sense to include estimates without interaction terms when the model DOES have interaction terms...
So although I want to include the main model estimates including interactions I face a problem - when estimating my pilot data the coefficient estimates became distorted in terms of positive and negative coefficient sign change. So for example if I estimate the interaction term price*discount; the discount coefficient goes from positive to negative, however the interaction coefficient itself is positive. In addition the interaction renders the variables statistically insignificant (which they were not before - although the sample only has 1040 observations). One would expect the discount to be positive on utility which poses the question of which coefficient to include for my final design for the mixed logit...

Proportionality is always good, in that case you just change the scale parameter and not the coefficients themselves.

Yes it is tricky which priors to take. You could use a model averaging approach in which both models are included in the efficiency criterion by specifying a model with and without interactions, in which they can also have different priors.

(Your guidance is hugely appreciated!)

The model averaging approach offers a great solution to the outlined challenges. (Actually the Bliemer, Scarpa and Rose paper on this very topic offers a very comprehensive overview of the methodology and theory.)

In terms of adopting this approach for my research:
- As you have previously advised in the literature (and this thread) the mixed logit specification comes with a significant computational burden and thus a design can be achieved effectively by applying the model averaging approach. However, how many models would I need to specify to generate an efficient design for rppanel if I need to incorporate uncertainty over (i) interaction terms (ii)dummy vs effects coding? If I specify models M1 for MNL, M2 for rppanel; how many more models should be outlined for appropriate inclusion of the other uncertainties aforementioned?

- Given the nature of the mixed logit framework, we know that that assumptions about the parameter population distributions can impact estimates. Greene&Hensher (2002) advise that for dummy variables a uniform distribution assumption can be useful. However I aim to set all my explanatory variables (3 continuous and 3 categorical with 4 levels each) to be randomly and normally distributed because the uniform specification appears to be unrealistic and restrictive in my case: but is this a logical conclusion? Does Ngene support other distributional specifications?

Cheers for the support!

Another query re: rppanel & model averaging approach => Ngene appears to ignore my constraint and require portion of the syntax when I attempt to generate a new design. The same syntax worked fine for my initial pilot survey design using the mnl. Now the generation process comes out with designs that ignore this aspect of the command. Having reviewed forum posts it appears this may just not be possible to impose at this stage? Any ways to overcome this?