Sample Size

[melanie]

Hi Michiel!

I was hoping you may be able to help me with my S-estimate/sample size in Ngene. I have obtained priors from STATA output from a pilot study of 21 respondents. I used the coefficient values, where if the coefficient was .5678 I would write the prior as 0.6 in the output below:

Design
;alts = Opioid, NSAID, optout
;rows = 60
;block = 5, minmax, noimprov(10 secs)
;con
;eff = (mnl,d)
;model:
U(Opioid) = b0 + b1.dummy[0.3]* LBP[2,1]
+ b2.dummy[0.2|0.3|0.3] * Comm[1,2,3,0]
+ b3.dummy[0.2|0.2|0.3] * Opioid[2,3,4,1]
+ b4[0.08] * Pain[2,1]
+ b5[-.005] * AEs[2,1] /
U(NSAID) = b6 + b7.dummy[0.6] * LBP[LBP]
+ b8.dummy[0.6|0.6|0.5] * Comm[Comm]
+ b9.dummy[0.2] * NSAID[2,1]
+ b10[0.007] *Pain[2,1]
+ b11[0.003]*AEs[2,1]
$

However I got a huge s-estimate, indicating that there must be something wrong with these priors?? I am not sure if you can see what I may have done wrong here?

Best wishes,
Melanie

[Michiel Bliemer]

The sample size estimates for all parameters are reasonably small, except for b5, b10, and b11. These parameters have very small prior values, which indicates that these attributes are not very relevant in making a choice and hence may not be statistically significant. It is difficult to say whether these small priors are due to the small sample size in your pilot or whether these attributes simply are not important (in which case these parameters will simply not be statistically significant in the final model).

I notice that you multiply Pain and AEs, which seem categorical variables, directly with a parameter. I think you need to dummy code them, e.g. b4.dummy[0.08] * Pain[2,1].

Michiel

[melanie]

Thanks Michiel!

Are you looking at the s estimates for each individual parameter, not the overall S-estimate? Maybe I am misinterpreting the values and how to assume a sample size from these numbers? I thought we calculated the overall S-estimate by the number of blocks to get the total sample size required? Also, with the priors does it matter how many decimal points are entered?

Regards,
Melanie

[Michiel Bliemer]

I look at the individual sample size estimates yes, they provide more information than just looking at the worst one.
Yes you need to multiply the sample size estimates with the number of blocks, in this case 5.
The number of decimal points does not matter much, priors are ballpark guesses.

You are estimating 18 parameters, that is quite a few, so you would expect that you would need quite a few respondents as well.

If your priors are correct, you will be able to estimate most parameters at a statistically significant level with 200-300 respondents, while you need 500-1000 to estimate 2 parameters at a statistically significant while, while a few parameters will likely not be statistically significant, even with more respondents.

I notice that you did not set any priors for b0 and b6, they are defaulted to 0. That is not appropriate, you need to set ALL priors in the model, otherwise your sample size estimates may be very wrong, so please also estimate the alternative specific constants in your pilot study and put them in as priors (or are they near-zero?).

Further, if you could generate a Bayesian efficient design with normally distributed parameters then your desighn would become more robust against prior misspecification. Bayesian priors can be derived from your pilot study as Normal(parameter estimate, standard error).

Michiel

[melanie]

Hi Michiel,

Thanks so much for your advice! It does seem like our priors from our pilot study are influencing the large S-estimate number. I have been playing around with the number of Bayesian priors and found that if I reduce the number it lowers the S-estimate, so perhaps not all priors should be coded as Bayesian? The best output I could get so far is with an overall s-estimate of 578, but then they range between 3.5 to 5475 for the individual Sp-estimates. How would you recommend I interpret these numbers? you recommend to go with the individual estimates but which number do I go with as there is one for each prior?

Best wishes

[Michiel Bliemer]

Sample size estimates are merely best guesses assuming that the priors are correct, but because priors are uncertain the sample size estimates are unreliable. Bayesian sample size estimates give a range of possible sample sizes and indicate the uncertainty about the sample sizes, and this can be a very big range. As you indicate, in your case perhaps between 3.5 and 5475. This is because your priors are not reliable enough to give a better sample size estimate, you would need a larger pilot study, 21 respondents is not much.

I use sample size estimates to see what parameters may be difficult to estimate, and see if I can improve by simplifying the choice experiment or by changing attribute level range etc, but do not use them to set my sample size for data collection. The sample size that I typically choose is dictated by my survey budget or available population. In most studies I aim for 1000 respondents and do a pilot study with 100 respondents to get priors.

Michiel

[melanie]

Many thanks!

Just one last question here, I did not code specifically to include interactions, as my understanding is this will increase the S-estimates? But does Ngene automatically assume any interactions unless I specify?

Regards

[Michiel Bliemer]

Ngene only optimises for the interaction effects when you specify them, but given that you have a relatively large number of choice tasks in your design you should be able to estimate all two-way interaction effects with the design (the design is just not optimised to estimate them).

Michiel