Requesting Input on Experimental Design

[Alizi]

Dear all,
I just finished the pilot survey and want to proceed to the final experiment.
I plan to analyze the data using a mixed logit model. The experiment includes three labeled alternatives, and based on the literature, I thought about 30 rows divided into five blocks such that each respondent gets six choice tasks.
This is the model I came up with:

Code: Select all

Design
;alts=evening, night, noon
;rows=30
;block=5
;eff=(rp,d)

;cond:
if(noon.work=0, noon.limitations=0),
if(noon.work=1, noon.day=0)


;model:
U(noon)=   b1[-0.837]                                                        +
           b2.dummy[-0.228]                     * work[1,0]                  +
           b3.dummy[n,0.201,0.822]              * day[1,0]                   +
           b4.effects[-0.323|-0.147]            * limitations[2,1,0]         +
           b5.dummy[0.010]                      * green[1,0]                 +
           b6[-0.015]                           * weeklycost[20:50:10]       /

 
U(evening)=b3                                   * day                        +
           b6                                   * weeklycost1[80:110:10]     /       


U(night)=  b8[-0.751]                                                        +
           b3                                   * day                        +
           b7.effects[-0.841|-0.946|-0.394]     * limitations2[5,4,3,0]      +
           b6                                   * weeklycost2[50:80:10]                       

$





I think the only coefficient I should treat as random is “day,” as it is the only coefficient with a significant SD in the pilot results. However, I can also treat “work” and “green” as random, which yields different designs with the following characteristics:
Day+work+green random: d-error= 0.871, S estimate=59663
Day+work random: d-error=0.51, S estimate=924
Day random (the design in the code above): d-error=0.31, S estimate=25886

My questions are:
1. Do I need to choose the design with the smallest d-error?
2. Is the d-error I get too high? What would be a reasonable d-error?
3. How should I interpret the S estimate in terms of sample size? Does 25K mean I should sample 25K people (N=25K)? Or is it the number of choice tasks (N*6=25K) or the number of observations (N*6*3)?
4. Is there anything I can do to improve this design?
5. BTW, how do you apply a non-significant coefficient from the pilot study as a prior in the final design? Should I use its non-significant coefficient or zero?

Many thanks in advance!

A.

[Michiel Bliemer]

1. You should not compare designs for different models, you can only compare D-errors for models with the same utility functions, the same model type, and the same priors. Note that your syntax does not include any random parameter, e.g. b1[-0.837] is a prior for a fixed parameter. Note that the RP model is not the appropriate model if you are giving 6 choice tasks per respondent, you need to use the panel version, RPPANEL. The RPPANEL model requires much more computation time. I generally recommend against optimising a design for mixed logit, I always use the MNL model to optimise a design and you can still estimate a panel mixed logit model afterwards.

2. D-error looks fine.

3. You need to look at the sample size estimate per parameter. Your prior for b5.dummy[0.010] is very small, this means that the attribute 'green' does not really influence behaviour and therefore a very large sample size would be needed to get a statistically significant estimate for this parameter. If most of your parameters have a sample size estimate that is reasonable then there should be no concern, not everything may come out statistically significant in the end, especially if attributes do not really matter to choice behaviour. You need to multiply the sample size estimates with the number of blocks, so times 6 to get the actual number of respondents needed.

4. Since constants are important in your experiment, please add ;con to your script.

5. I generally use Bayesian priors after a pilot study, something like b[(n,0.5,0.3)] where 0.5 is the parameter estimate and 0.3 is its standard error. This parameter is not statistically significant, but 0.5 is still the best estimate and using a Bayesian prior accounts for its unreliability. If the parameter has an unexpected sign, then I could adjust the prior to a small value with the correct sign and apply the same standard error. If you use Bayesian priors, you need to also change the ;eff command and specify ;bdraws.

Michiel

[Alizi]

Thank you, Michiel, for your insightful and helpful response.
I greatly appreciate your suggestions! I have implemented all of them, and the design has been significantly improved.
A.