Replacing insignificant priors

[Liz]

Hi, thank you for a great piece of software. I have been reading the forums and Ngene manual for a while, but as this is the first time designing with Ngene, I just want to check I have understood correctly before I launch this survey.

I have a labelled experiment with three alternatives plus a no choice option. I have carried out two pilots now, the second after I refined my attributes and alternatives. The priors from my second pilot (45 people) were ok but two of the attributes (source and availability) were not significant. I ran the code with the real priors and SEs in anyway and it resulted in S estimates that were huge (>3000), so in the code below I have removed these and replaced them with 1 and -1, based on the direction for the priors in the pilot.

My question is: a) is my coding right; and b) is this the right approach to take? I could probably estimate based on my knowledge, and the literature what the rough preferences would be for source and availability, but I am not sure how to do this alongside the actual priors from the the pilot that I already have e.g. I know from the literature that the people I hope to target with this experiment tend to prefer the '0' source level, but my pilot suggested (albeit insignificant) preferences for '1'. Any suggestions?

I am also not sure how to deal with the b1 constant in a labelled experiment - the below code uses the intercept value (there was a strong preference for alt1) but is that correct?

Code: Select all

Design
;alts = alt1,alt2,alt3,alt4
;eff = (mnl,d,median)
;rows = 25
;block =4
;cond :
if(alt2.form=0,alt2.source=0),
if(alt2.form=0,alt2.price=[500,1000])
;model:
U(alt1) = b1[(n,-1.130, 0.149)] + source[(n,1,0)]*source[0,1] + avail[(n,-1,0)]* avail[0,1] + form[(n,-0.296,0.116)]* form[0,1]+ price[(n,-0.001,0)]*price[10,100,250,500,1000]/
U(alt2) =b1+source*source +avail* avail + form* form+ price*price/
U(alt3) = b1+ source*source +avail* avail + form* form+ price*price
$


Result
MNL efficiency measures

Bayesian
Fixed Mean Std dev. Median Minimum Maximum
D error 0.016593 0.016747 0.001461 0.016602 0.014037 0.021233
A error 0.359925 0.363356 0.031812 0.359745 0.304172 0.460436
B estimate 20.829165 20.96379 4.601412 20.845549 10.741518 31.983946
S estimate 18.669063 45.922478 126.870047 18.761615 4.734408 1355.678791

Prior source avail form price
Fixed prior value 1 -1 -0.296 -0.001
Sp estimates 1.875208 2.019829 18.669063 2.88119
Sp t-ratios 1.431302 1.37911 0.453623 1.154703
Sb mean estimates 1.89193 2.037997 45.922478 2.905603
Sb mean t-ratios 1.428911 1.376896 0.454238 1.152596


I have also been reading about the designs that use varying scenarios, and this is great as I was originally thinking of running two experiments each with a different scenario. I have not tried this yet as I want to sort out the priors first, but would it be relatively simple to add two scenario options into the above code, or would it require a bit more thought than that?

[Michiel Bliemer]

There are several questions throughout your post (perhaps you can number them or something next time so it is easier to respond to). I will try to answer them below.

1. I would recommend to always include the ;bdraws command to indicate the number of draws since the default may not be enough. I suggest using ;bdraws = sobol(500). Further, since you have attribute with 2 levels and 5 levels, I would recommend generating a design with 30 rows such that you have attribute level balance. Also, you want to make sure that the number of rows is divisible by the number of blocks. Using ;block = 5 with ;rows = 30 will give you 6 choice tasks in each block. Finally, if alt1-3 are truly labelled, then you will need an alternative specific constant for each alternative. Now you are using b1 for all three alternatives, while I would expect three different constants.

2. Priors should be set very carefully, and if they are unknown they should be set close to zero. It is usually not appropriate to use -1 and 1 for priors that are not statistically significant, this may mess up the entire model efficiency. Even if you could not estimate statistically significant parameters in the pilot study, you can still use the values that come out of estimation, or set them to very small values, i.e. -0.00001 and 0.00001. Values very close to 0 are always safe. The S-estimates should be inspected per parameter, you can ignore the S-estimates for parameters that were not statistically significant or are set close to zero because they are meaningless. S-estimates are only meaningful if you sufficiently trust the value of the prior.

3. See my comment in 1, I believe you need to specify three different constants.

4. You can add scenario variables as an extra attribute in the utility functions, either as a main effect or as an interaction effect.

Michiel

[Liz]

Many thanks for this detailed reply - apologies for asking so many questions in one post!

This all makes sense - two more follow up questions though:

1) Great - I will put in the priors again from my pilot. In this case though, how do I know what my target sample size should be, if the S estimates are not all reliable? Should it be the highest value for the parameters I do trust?

2) My alternatives 1-3 are truly labelled. I ran an MNl on my pilot data to get the priors for the final design, but, while I can get alternative specific constants for alt1 and alt2, I cannot get one for alt3 as SPSS tells me it is redundant. I have tried everything I can think of but it is always redundant. SPSS issues are not your problem (!) so my question is, in a case where I cannot get a value, should I set the alternative specific constant for alt3 to 0 for the final design?

Many thanks again for all of your help.

[Michiel Bliemer]

1.
Note that sample sizes provided by Ngene are MINIMUM sample sizes you will need (for obtaining statistically significant parameters), you usually want larger values. Also it does not account for covariates such as income, age, gender, etc, which usually require many more respondents.

There exist no other sample size calculations, most of the time people just based sample size on the amount of money they have (100 respondents, 1000 respondents). If you are sampling consumers, then you usually can get a lot. If you are sampling a very specific audience, then you want to get your priors are reliable as possible to make an efficient as possible survey.

2.
Are you sure that you have put 4 alternative into the estimation software? Your fourth alternative should be U(alt4) = 0. If you put in this fourth alternative, you should be able to estimate 3 constants.

[Liz]

Many thanks for the help with this last year, we put it on hold for a while but will be implementing it soon. I just had one follow-up question from your reply last year: why do we use sobol here in particular? I ask because a colleague suggested that Halton draws should always be used as a default, and I wasn't sure how this decision would be made. I have read the manual, about how Sobol draws account for some problems with Halton draws, but I am still not sure whether this is due to some characteristic of the design being used?

Thank you!

[Michiel Bliemer]

Halton draws would also work but they are known to be problematic in higher dimensions, see for example this paper:
https://www.sciencedirect.com/science/article/pii/S0191261502000905

Instead of shuffling or shifting Halton draws, Sobol draws do not have the issues that Halton draws have. Furthermore, a paper presented at the International Choice Modelling Conference in 2017 found that Sobol draws are superior to other existing quasi-random draws (such as Halton draws) in model estimation.

With respect to Bayesian efficiency and quasi-random draws, we also found that Sobol draws outperform Halton draws:
https://www.sciencedirect.com/science/article/pii/S1755534513700241

All evidence seem to point to the fact that Sobol draws are better than Halton draws, they are just as easy to use as Halton draws without the issues. This is why I always recommend using Sobol draws when calculating median Bayesian D-errors and I recommend Gaussian quadrature when calculating mean Bayesian D-errors (which is an even smart technique but can only be used to calculate means, not medians). If you prefer using Halton draws that is also fine, but I disagree with your colleague that they should be used as the default, this was maybe true 15 years ago but the literature has moved on.

Michiel