modify the type of variable in the utility for design

[SupphaCH]

Dear experts,

I'm in the process of planning an experimental design in Ngene, and I have a couple of questions related to creating an efficient design. I'd appreciate your insights on the following:

(1) Can I modify the type of variable in the utility function from the pilot study design to the main study design? For instance, in the pilot study's design of labelled alternatives, all variables are generic. However, I'm considering changing some variables in the main design to include a mix of generic and specific variables (with the number of attributes and levels remaining the same). Would such a change impact the efficiency of the design?

(2) When it comes to obtaining priors from the pilot study, should we aim to improve model estimation until all parameters are statistically significant, or is it acceptable to remove certain attributes from the model? If we encounter parameters from a pilot study that are not statistically significant and have incorrect signs, how can we address this issue? Can we still use these parameters for the experimental design, or are there alternative solutions available?

I'm grateful in advance for any assistance or guidance you can provide on these matters.

Thank you,

[Michiel Bliemer]

(1)
When you say "generic variables", do you mean "generic parameters"? If you are changing some parameters from generic to alternative-specific, then you need to estimate more parameters and therefore design efficiency will change. If "generic variable" means "generic attribute" and you are changing the levels of the variable, then also the design efficiency will be impacted. A design is efficient assuming (i) utility functions, (ii) model type, (iii) priors. If any of these inputs changes, then the design will lose efficiency. How much efficiency is lost is case specific, but the more you deviate from the assumptions under which you generated the design, the more efficiency will be lost.
I would not worry about it too much though, efficiency of a design is only a desirable but not a necessary condition to estimate models. A design for a pilot study is often optimised using uninformative priors, but the actual parameter values will differ from the assumed priors and therefore some efficiency will be lost anyway. I would make all changes that you deem necessary and generate a new design for the main study.

(2)
The sample size of a pilot study is often small, which means that certain parameter estimates may not be statistically significant. I would not remove attributes associated with parameters that are not statistically significant in the pilot study, as they may become significant with a larger sample size in the main study. I usually use 10% of the total sample for the pilot study, estimate the model with all attributes, and use whatever parameter estimates come out. Even if a parameter estimate is not statistically significant, it is still the best guess for the value you have, even if it is unreliable (i.e. large standard error).
For statistically significant parameters I generally assume normally distributed Bayesian priors: (n,mean,stdev), where mean = parameter estimate and stdev = standard error.
For parameters that are not statistically significant, the standard error is relatively large and therefore a normally distributed Bayesian prior may have a very wide distribution. In that case, draws from the distribution may become quite extreme. In that case it would be best to use the median D-error instead of the mean D-error to avoid that outliers negatively impact the design. In Ngene you would use ;eff = (mnl,d,median) and ;sobol = bdraws(...). Alternatively, you can consider using a (bounded) uniformly distributed Bayesian prior where the parameter estimate is the midpoint and the range is chosen such that it reflects the standard error.
If parameters have the wrong sign, then I would manually correct that in the prior. For example, if a parameter estimate is -0.2 but it should be positive, then I would use a uniformly distributed Bayesian prior (u,0,..) that is bounded from below by zero, where the upper bound is chosen such that it is reasonable (please make sure that it is not too far away from zero; if in doubt, close to zero is safe, too far away from zero could mess up your design as it would make the attribute dominant).

Experimental design is an art, not a science, so do not be afraid to make manual changes if needed.

Michiel

[SupphaCH]

Hi Michiel,

Thank you so much. It’s quite clear for me.
Another question about S-estimation in Ngene.
I’m wondering about fixed- and mean- value for a sample size. What is the difference?
What number should I need to use for a minimum sample size for the main study?

[Michiel Bliemer]

Sample size estimates can be based on local (fixed) priors or on Bayesian priors.

The "fixed" S-estimate is the sample size estimate assuming only the mean of the Bayesian distribution, whereas the "mean" S-estimate is the expected sample size over draws from the entire Bayesian distribution. Each draw from the distribution will result in a different S-estimate and the "mean" is therefore simply the average over all draws. If there are outliers in the draws then mean Bayesian S-estimate could become quite large.

Michiel