choice-metrics.com

Dear Ngene community
I am running the efficient design below but my question is more general.
According to the manual, "Many syntax runs will execute for a long time or indefinitely. A syntax run can be stopped at any
stage by selecting the Stop menu item.". But it doesn't say when stopping is appropriate. What stopping rule should be used ?
My intuition would be that this is when the improvement in the D-error is low, but what is low ?
Thanks for your help
Marianne

Design
;alts=GA*, GB*, SQ
?place an asterisk next to the names to prevent from having i) within choice task alternative repetition, ii) strict attribute level dominance and iii) choice task repetition given attribute bundle ordering

;rows = 12
?make sure rows>nb of parameters beta/(nb of alternatives-1)+ nb rows multiplies the number of levels of all attributes (2,4,3,6)

;block = 2

;eff = (mnl,d)
;model:
U(GA) = b1.dummy[-0.206]*A[1,0] + b2.dummy[0]*B[1,0] +b3[0.033]*C[65,80,90,95] +b4[-0.156]*D[1,2,3,4,5,8] /
U(GB) = b1*A + b2*B + b3*C + b4*D /
U(SQ) = b0[-1]
$

It is a matter of preference, but there is little to be gained if the D-error (or another efficiency measure) only decreases by 1% after 1 hour of evaluating designs. I usually aim to evaluate around 1 million designs (you can see this number in the bottom of the output window) and then stop the script as it is unlikely that Ngene will be able to find a better design after evaluating this many possible designs. This may mean running the script overnight for example, but if you intend to make further changes to your script then of course you would not run it for so long. One million designs may take a while when you have Bayesian priors so in that case you may decide to look at the improvement after each 10,000 design evaluations or so. If this improvement is marginal, then you likely have found a sufficiently efficient design already. You will hardly ever be able the find THE MOST EFFICIENT design, since this would require evaluating all possible designs (and since there are billions of them, this is not computationally feasible in most cases).

We are working on a default stopping criterion in Ngene, which we aim to implement in our release early next year.

Michiel

Thanks a lot Michiel.
Looking forward to the forthcoming default stopping criterion in Ngene to help us saving energy !