Revising attributes/levels after collecting pilot data?

[Dana]

Hello,

I am part of a team who has designed a discrete choice experiment with five attributes, four of which have three levels and the one (WTP) has five. We administered a pilot study (n = 28) that included two blocks of nine choice scenarios (total of 18 choice scenarios) for respondents to evaluate. Using multinomial and random parameters logistic regression models to analyze pilot data, we found that two of our attributes were performing quite poorly. To improve our model for the final survey instrument, we are considering removing one of these attributes and revising the second problematic attribute to widen the range between each level. However, we are using the pilot data to re-estimate a D-efficiency design (using Ngene) and are wondering about the implications of these revisions on the final design. Specifically, we are seeking clarification on whether this would be possible and, if so, advisable. Alternatively, we are considering estimating a new design, forgoing the opportunity to improve the efficiency of our original design with prior estimates.

We would very much appreciate your insight on the points above and look forward to hearing from you soon.

Kind regards,
Dana Johnson

[Michiel Bliemer]

Hi Dana,

Whether you can use priors based on estimated from a pilot study depends on whether they are sufficiently reliable/stable. If you obtain coefficients that have the wrong sign, then I would not use those coefficients as priors. If you obtain coefficients that are not statistically significant but have the right sign, then you can use those coefficients as Bayesian priors, where the standard deviation of a normally distributed Bayesian prior indicates the level of unreliability (i.e., the standard error of the coefficient). Therefore, in most cases I think that you will be able to generate a Bayesian efficient design based on priors obtain from your MNL pilot estimates. I would not estimate a random parameter model based on such a small sample size.

Michiel

[Dana]

Hi Michiel,

Thank you very much for your swift response. It is really helpful to hear that we could use these priors to estimate a Bayesian efficient design. It is also helpful to hear your insights on whether to use our pilot study coefficients as priors. There were a few coefficients that did not have the correct sign. These are the problematic attributes that we are considering revising or removing. I have pasted the Nlogit output from our MNL model below - we are considering removing ATT_Win and revising the levels for ATT_OST (because these levels were not spread very far apart, we are planning on widening the range for the final model). If we revise the levels of ATT_OST and remove ATT_Win, would we still be able to use these coefficients (committing ATT_Win) as priors to generate a Bayesian efficient design? Thank you very much for your insight! I am very new to choice modeling and this information is tremendously helpful.

Kind regards,
Dana

Output from MNL Model in Nlogit :
|-> NLOGIT
;LHS = CHOICE
;Choices = 1,2,3
;RPL
;Fcn = Att_Mos(n), Att_Fre(n), Att_OST(n), Att_Win(n), Att_WTP(ln), one(n) ;Model: U(1,2) =
+ ATT_Mos * ATT_Mos
+ ATT_Fre * ATT_Fre
+ ATT_OST * ATT_OST
+ ATT_Win * ATT_Win
+ ATT_WTP * ATT_WTP
/
U(3) = one $
Iterative procedure has converged
Normal exit: 6 iterations. Status=0, F= .2265967D+03

-----------------------------------------------------------------------------
Start values obtained using MNL model
Dependent variable Choice
Log likelihood function -226.59669
Estimation based on N = 252, K = 6
Inf.Cr.AIC = 465.2 AIC/N = 1.846
---------------------------------------
Log likelihood R-sqrd R2Adj
Constants only -231.9123 .0229-.0009
Note: R-sqrd = 1 - logL/Logl(constants)
Warning: Model does not contain a full
set of ASCs. R-sqrd is problematic. Use
model setup with ;RHS=one to get LogL0.
---------------------------------------
Response data are given as ind. choices
Number of obs.= 252, skipped 0 obs
--------+--------------------------------------------------------------------
| Standard Prob. 95% Confidence
CHOICE| Coefficient Error z |z|>Z* Interval
--------+--------------------------------------------------------------------
ATT_MOS| .00670 .00626 1.07 .2849 -.00558 .01897
ATT_FRE| .01525** .00630 2.42 .0155 .00290 .02761
ATT_OST| -.00211 .00943 -.22 .8230 -.02058 .01637
ATT_WIN| -.01282 .01184 -1.08 .2789 -.03604 .01039
ATT_WTP| -.00401** .00200 -2.00 .0450 -.00794 -.00009
Constant| -1.70277*** .43604 -3.91 .0001 -2.55739 -.84815
--------+--------------------------------------------------------------------
***, **, * ==> Significance at 1%, 5%, 10% level.
Model was estimated on May 12, 2020 at 11:55:10 AM
-----------------------------------------------------------------------------

[Michiel Bliemer]

For example, ATT_OST has a coefficient of -0.00211 and a standard error of 0.00943, such that the coefficient is not statistically significant with the given sample size. This could be a sample size issue or could be because your levels are not wide enough (as you mention). If this coefficient is of the wrong sign, you could either set the coefficient to zero or a very small positive value (e.g. 0.00001) to indicate it is positive in case you want to check for dominant alternatives in Ngene (which requires specifying signs). So in that case, you would have the following Bayesian priors:

(n,0,0.00943) or (n,0.00001,0.00943) indicating that there is a lot of uncertainty about this coefficient.

Changing the levels mainly changes the standard errors, it should not affect your coefficient much so you can still you use your coefficients as a Bayesian prior.

Removing attributes generally means that other parameter estimates also change, so the impact on the other coefficients is unclear, but I think that it is reasonable to assume that they are similar and therefore use the existing coefficients for Bayesian priors.

Note that Bayesian priors are merely BEST GUESSES for the parameter estimates, it is not an exact science, so even if your priors are not entirely correct it should be fine. The only time where it goes wrong is if your priors are very bad, e.g. wrong signs, completely wrong magnitudes, but given that your coefficients come from a pilot study, this is probably the best you can do.

Michiel

[Dana]

Dear Michiel,

Thank you very much for this very insightful information. We have been able to move forward with generating our experimental design confidently thanks to your post. Please accept our sincerest gratitude and we look forward to connecting more on choice modeling related conversations through this forum.

Have a lovely weekend.

Kind regards,
Dana