Large experimental designs

[Doug Clover]

Dear All

I am hoping to use the efficient design functionality in NGENE to implement an ambitious SC design. At this stage I envisage a design with up to 12 alternatives and 19 attributes.

Has anyone used NGENE to eff design a similar sized SC experiment? I have attempted a couple of mock up experiments using parameters from the literature to determine probable sample size (mnl, s, fixed). It ran successfully but did not even achieve one iteration even after 12 hours. Is this inevitable for a design of this size or is it something to do with my specification?

Any thoughts would be appreciated.

Doug

[Andrew Collins]

Hi Doug,

So long as you are just optimising on an mnl model, there should be no problems with a design of that dimension. I suspect the problem might be that the combination of priors is leading to very unbalanced choice probabilities, with some alternatives never chosen, and this is leading to a singular fisher matrix. In this case the design is thrown out. When you run the design, does the 'Current number of invalid designs' increase at the bottom right of the output window?

If this is the cause, then you may need to examine the combination of priors and design levels carefully. You could perhaps start with 2 alternatives, and progressively add in alternatives, keeping a close eye on the choice probabilities.

Let me know if this helps.

[Doug Clover]

Dear Andrew

No current number of invalid designs stays on 0. The only indication that anything is happening is that in the session history panel the status is given as running and the same at the bottom left corner.

I have managed to get some progress with 3 alternatives and 4 attributes (but the sample estimates have been somewhat daunting for a design of that size). But when I jumped to 3 alternatives and 10 attributes it stalled again. I will be more patient and progress more slowly.

I am just doing this exercise to get some idea of what the sample size might be for budgetary reasons. If the survey scale seems manageable without having to simplfy the expeeriment too much I intend to do an orthogonal design experiment with a pilot group to derive my priors for the design of my full survey.

Doug

[Andrew Collins]

So long as it is a fixed mnl design, the 3 alts by 10 attribs should not be a problem, I suspect there is something else going on. I am happy to have a look over the syntax and try and work it out, if you are comfortable sending it to me (it could be by email or private post through the forum). Ngene should detect it, but have constants been specified for all alternatives? That would cause problems.

[Doug Clover]

Dear Andrew

Thank you for your kind offer and I appreciate it a lot. I may impose on your time in the future, but at this time I would like to persevere at little longer before I start bothering you or John.

Just to give you some more background on my intentions. I want to estimate an rp panel model, but from the DC design course I know that this type of model would never resolve (well maybe for a year of Sundays). Therefore, I intend to use the approach suggested by Michiel where you optimise on a mnl model, but have NGENE also design a rp panel. This approach was suggested during the session on model averaging.

Doug

[Doug Clover]

Dear Andrew


I feel that I will be popping up here frequently over the next few months.

The purpose of my experiment is to forecast choice shares of different types of new vehicle technologies as prices change and technical performance improves over time. As I have mentioned in an earlier post the first cut of the design is large and will need to be reduced to be more practical.

At this stage I am still trying to get some idea of minimum sample size for the bells and whistles version of the design.

My question relates to the role of constants in the design process. I note that in the manual states that the normal practice for NGENE is to exclude the constants. If constants are excluded is a prior parameter estimate for the constant necessary? What effect, if any, does the accuracy or inaccuracy of the estimated prior for the ASCs have onthe design? Shouldn’t I just set them to zero, at least for the first design, which will be used to develop the pilot survey questionnaire? By the way in a previous message I said that I was going to use an orthogonal design for the pilot survey, but I am afraid poor old NGENE couldn't manage it. I suspect I ran up against the 150,000 limit.

Cheers Doug

[Michiel Bliemer]

Dear Doug,

I agree with Andrew that it should be no problem to optimize your design even for larger numbers of alternatives and attributes for the MNL model. So as long as ;eff = (mnl,d) or something is your efficiency measure, that should be fine. Once you retrieve your MNL optimized design, you can evaluate it as if you were estimating a mixed logit model. If it still runs slow, remove anything referring to random draws or random parameters in your model to make it completely MNL, just to make sure there is no simulation involved.

An orthogonal design for such a large number of alternatives and attributes is most likely very hard to find indeed. But of course you can always start with an efficient design in which you just set all priors equal to zero (no information). This is in some ways similar to determining a (near) orthogonal design, although you can keep the number of choice situations limited.

With respect to your question on constants, yes you have to specify the priors for the constants, even if you do not care about their standard errors in estimation. The constants influence the values of the utility functions, and therefore also the logit probabilities, which again influence the asymptotic standard errors for all other parameters. So the priors of the constants could potentially have a large influence on the efficiency of your design. In case you have no information on them, indeed you can set them to zero. Or use some Bayesian priors to take into account the uncertainty about the prior, although keep the number of Bayesian priors limited in order to avoid long computations times.

By the way, may I suggest starting a new thread on the forum for different topics (not relating to your initial question on "large experimental designs"? Then other users can probably find answers to similar questions more quickly.

Feel free to popping up frequently on here, hope we can be of any help.

---Michiel

[Doug Clover]

Dear Michiel

Thank you for the reply. It has helped me a lot.


I will also make sure if I have otheer questions or comments start new message threads.

cheers Doug