choice-metrics.com

If I generate an S-estimate of say, 1,000, but only can afford 300 observations, does this mean I need to find a new design?

The S-estimate is an indication of the sample size you would need to obtain a statistically significant parameter estimate at the 95% confidence level, and they assume that your parameter priors are correct. This last assumption is of course very strong, so if your priors indeed turn out to be correct, then you would indeed need 1000 respondents to estimate this parameter with a t-ratio of above 1.96. This would be a lower bound, so you will likely want to have more respondents.

However, if your priors turn out to be very different from the true parameters, then the S-estimate could be very different. For example, if you thought that a parameter is 0.1 and it turns out to be 0.3, then you may need a much lower sample size, since estimating parameters with a small contribution to utility (i.e. a coefficient of 0.1) is much more difficult than estimating parameters that have a larger contribution to utility (i.e. a coefficient of 0.3). Typically, dummy/effects coded parameters and standard deviations in random parameters are difficult to estimate and require more respondents.

Note that generating a design in which the S-error (the maximum of all S-estimates) is lower than 300 may not exist. So it is important to check how certain you are about the parameter priors you have used, and if you think they are fine, then you may not be able to obtain all statistically significant parameter estimates with 300 respondents, and the individual S-estimates will tell you which parameters are particularly difficult to estimate. Changing the range of the attribute levels may help, so if you have b1[0.1] * A[2,3,4] you may want to change this to b1[0.1] * A[1,3,5] if this makes sense. Using a wider range significantly decreases the sample size you need. Of course this is not possible with dummy/effects coded parameters.

Michiel

Dear Michiel,

just a follow up on the topic of S-estimates. I have designed a D-efficient experimental design (see below) where the priors are informed by expert opinion, but unfortunately we don't have literature or the possibility to do many pilots to estimate a model and validate the prior values. I wanted to check whether I should be worried by the high S-estimate that I'm getting from the design - I've read that high S estimates could be driven by an inappropriate choice of priors. In this context, what would you advise to do: carry on with this design or apply some modifications? I also have to say that, because it will be an expert survey, we are very unlikely to get more than 80 or so respondents in the final data collection, so we definitely won't be able to achieve big sample sizes.

Many thanks in advance for the attention you will give to my post and for any advice.

Best wishes,

Michela

Design
;alts=reclaimed*,new
;rows=24
;block=3,minsum
;eff=(mnl,d)
;bdraws=sobol(2000)
;con
;store=all
;model:

U(RECLAIMED)=b1.dummy[(n,0.05,0.05)|(n,0.03,0.03)|(n,0.02,0.02)] * CARBON[3,2,1,0]
+b2.dummy[-0.04|-0.02] * APPEARANCE[2,1,0]
+b3[-0.001] * COST[50,200,350,500] /

U(NEW)= new[-1]

;cond:
if(reclaimed.APPEARANCE=0,reclaimed.COST=[350,500])$

The priors are very conservative, your attributes all have a quite small contribution to utility (price has the largest contribution, the other attributes have very minor relevance given these priors) and therefore the sample size requirement seems very large. I think that your priors are too low and that your sample size estimates are therefore inflated. There is not really anything else you can do, I think you need not worry too much about the survey, it looks fine and I suspect that you will be able to estimate most coefficients to a sufficient statistical significance level as long as people are trading off on all attributes (i.e., as long as price is not dominant, otherwise you may need to somewhat decrease its range). If you could do a pilot study with 10 responents that would tell you whether people make trade-offs on all attributes.

Michiel

Thanks Michiel, this is very helpful (and reassuring). Sorry, I just realised that the code that I copy-pasted in my earlier message was a trial version in which I divided all non-cost attribute coefficients by 10. So, in the final design all non-cost coefficients will be 10 times higher, which should help in terms of balancing out the contribution of all attributes to utility. I still get high S-estimates even when using these higher coefficients. However, it is reassuring to know that this won't matter as long as respondents make trade-offs. Based on a very small number of pilots that we managed to carry out, there are good signals that respondents make trade-offs, so fingers crossed this will hold also with the main sample of respondents.

Thank you.

Best wishes,

Michela

You may want to consider this syntax, in which you have 2 alternatives of reclaimed. This will allow more trade-offs.

Code: Select all

Design
;alts=reclaimed1, reclaimed2,new
;rows=24
;block=3,minsum
;eff=(mnl,d,mean)
;bdraws=gauss(5)
;con
;store=all
;model:

U(RECLAIMED1)=b1.dummy[(n,0.5,0.25)|(n,0.3,0.15)|(n,0.2,0.1)] * CARBON[3,2,1,0]
+b2.dummy[-0.4|-0.2] * APPEARANCE[2,1,0]
+b3[-0.001] * COST[50,200,350,500] /

U(RECLAIMED2)=b1 * CARBON[3,2,1,0] + b2 * APPEARANCE + b3 * COST /

U(NEW)= new[-1]

;cond:
if(reclaimed1.APPEARANCE=0,reclaimed1.COST=[350,500]),
if(reclaimed2.APPEARANCE=0,reclaimed2.COST=[350,500])
$

Michiel

Or evem better, this syntax, which avoids dominant alternatives:

Code: Select all

Design
;alts=reclaimed1*, reclaimed2*,new
;rows=24
;block=3,minsum
;eff=(mnl,d,mean)
;alg = mfederov(candidates = 800)
;bdraws=gauss(5)
;con
;store=all
;model:

U(RECLAIMED1)=b1.dummy[(n,0.5,0.25)|(n,0.3,0.15)|(n,0.2,0.1)] * CARBON[3,2,1,0]
+b2.dummy[-0.4|-0.2] * APPEARANCE[2,1,0]
+b3[-0.001] * COST[50,200,350,500](4-8,4-8,4-8,4-8) /

U(RECLAIMED2)=b1 * CARBON[3,2,1,0] + b2 * APPEARANCE + b3 * COST /

U(NEW)= new[-1]

;reject:
reclaimed1.appearance = 0 and reclaimed1.cost = 50,
reclaimed1.appearance = 0 and reclaimed1.cost = 200,
reclaimed2.appearance = 0 and reclaimed2.cost = 50,
reclaimed2.appearance = 0 and reclaimed2.cost = 200
$

Michiel