Reference alternative in an orthogonal design

This forum is for posts covering broader stated choice experimental design issues.

Moderators: Andrew Collins, Michiel Bliemer, johnr

Reference alternative in an orthogonal design

Postby Rim » Wed Dec 19, 2018 10:47 pm

Hello!
I'm trying to do an orthogonal design for a choice experiment where the participants choose between 4 mobility alternatives. I want to fix one of the 4 alternatives as a status-quo for the individual but I don't know if it is correct to define a reference scenario in an orthogonal design? If it is possible, how to program it in my case? (I only find the answer for my question when using an efficient design)

The following is the preliminary design I'm using:

Design
;alts = Car, PT, Bicycle, Walking
;rows = 72
;orth = sim
;block=8
;model:
U(Car) = b1 + b2 * Time[-25,0,25] + b3 * Cost[-25,0,25] + b4 * Ind_mortality[0.7,0.8,0.9] + b5 * Coll_mortality[0.7,0.8,0.9] /
U(PT) = b2 * Time + b3 * Cost + b4 * Ind_mortality + b5 * Coll_mortality + b6 * Frequency[5, 10, 20]/
U(Bicycle) = b2 * Time+ b4 * Ind_mortality + b5 * Coll_mortality /
U(Walking) = b2 * Time+ b4 * Ind_mortality + b5 * Coll_mortality
$


Thank you in advance :)
Rim
 
Posts: 6
Joined: Tue Dec 11, 2018 8:32 pm

Re: Reference alternative in an orthogonal design

Postby Michiel Bliemer » Thu Dec 20, 2018 8:23 am

An orthogonal design does not optimise for the specified model, so you can simply leave out the reference alternative and create an orthogonal design without it. In your survey, you simply add your reference alternative. Note that your data will not be orthogonal because of the reference alternative. Note that there is NO benefit in using an orthogonal design for choice experiments.

I generally recommend using an efficient design with zero priors, which creates a design that maximises information and accounts for the reference alternative and has much more flexibility.

Michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm

Re: Reference alternative in an orthogonal design

Postby Rim » Thu Dec 20, 2018 7:31 pm

Thank you for your reply! I will follow your recommendations :D
Rim
 
Posts: 6
Joined: Tue Dec 11, 2018 8:32 pm

Re: Reference alternative in an orthogonal design

Postby Rim » Fri Jan 11, 2019 7:13 pm

Hi, sorry for bothering you again!
I followed your recommendation concerning using an efficient design with priors=0 and the following is the code I created (the status-quo for the respondent is using the car as a mean of transport):

Design
;alts= Car, PT, Bicycle, Walk
;rows=12
;eff = (mnl,d)
;block=3
;model:
U(Car) = b1[0] * Time.ref[10]
+ b2[0] * Cost.ref[5]
+ b3[0] * Ind_mortality.ref[1.3]
+ b4[0] * Coll_mortality.ref[0.5]/

U(PT) = b1 * Time.piv[-25%,0%,25%]
+b2 * Cost.piv[-25%,0%,25%]
+ b3 * Ind_mortality.piv[0,1,2]
+ b4 * Coll_mortality.piv[0,1,2]
+ b5[0] * Frequency[5, 10, 20]/

U(Bicycle) = b1 * Time.piv[-25%,0%,25%]
+ b3 * Ind_mortality.piv[0,1,2]
+ b4 * Coll_mortality.piv[0,1,2]/

U(Walk) = b1 * Time.piv[-25%,0%,25%]
+ b3 * Ind_mortality.piv[0,1,2]
+ b4 * Coll_mortality.piv[0,1,2]
$


However, running the code on Ngene, it takes a long time making evaluations and it does not converge to a final result. Reading the manual, I found that if a non-bayesian mnl program does not generate quickly this means that I have a problem of specification but I can't see where the error is in my code?
I even tried to run an example code from the manual to check whether the problem is coming from the software itself and it is taking too long to generate the final result too (almost an hour running it is still the same status). This is the code I used from the manual:


Design
;alts = alt1, alt2, alt3
;rows = 12
;eff = (mnl,d)
;model:
U(alt1) = b1[0.6] * A.ref[2] + b2[-0.1] * B.ref[5] /
U(alt2) = b1 * A.piv[-1,0,1] + b2[-0.2] * B.piv[-25%,0%,25%] /
U(alt3) = b1 * A.piv[-1,0,1] + b2[-0.2] * B.piv[-25%,0%,25%]
$


Could you please check my code for mistakes?
Thank you for your time and help.

Regards,
Rim.
Rim
 
Posts: 6
Joined: Tue Dec 11, 2018 8:32 pm

Re: Reference alternative in an orthogonal design

Postby johnr » Sat Jan 12, 2019 11:38 am

Hi Rim

The software doesn't converge unfortunately as when estimating a model. In estimation, we have algorithms to locate the maximum LL function (e.g., Newton-Raphison). When generating a design, whilst we have an objective function (say minimise the D-error), there is no algorithm to continually improve the design iteratively as you do in estimation with the LL function (unless you allow for continuous non-discrete levels), or you make a number of very restrictive assumptions. Hence, most algorithms test a set of random designs (some in smatter ways than others) as there are probably billions of billions of billions of possible designs for any given problem. Only for very small design problems can you check the objective function for all possible designs. Hence, the software will keep running (checking more designs) until either a) the user stops it and selects the best design found at that time, b) we finally as a species destroy the world and hence the computer running the software, or c) the sun finally explodes and takes with it the earth (and also the computer).

This is why we talk about efficient rather than optimal designs. Unless you check the objective function for every possible design, you won't know if it is optimal - just that it is efficient.

Your original code works perfectly!

John
johnr
 
Posts: 168
Joined: Fri Mar 13, 2009 7:15 am

Re: Reference alternative in an orthogonal design

Postby Michiel Bliemer » Tue Jan 15, 2019 11:42 am

Finding the optimal (most efficient) design with the lowest D-error is a so-called NP-hard problem, i.e. the time it takes to find an optimal design increases exponentially with the dimensions of the design. There are typically trillions of possible experimental designs and evaluating all of them to find the best design takes very long as John points out.

As you will see in the output window in Ngene, the D-error initially decreases quickly (from 0.04 to 0.03) but after some time the D-error only marginally improves (e.g. from 0.0283 to 0.0282). One can argue that a design with D-error 0.0282 is sufficiently efficient and you simply manually stop the algorithm once the D-errors do not decrease much anymore. The D-error of the optimal design is unknown, but will be around, say, 0.0280. Ngene will eventually find the optimal design, but it could take days, months, or even years. The benefit of using an optimal design with a D-error of 0.0280 instead of an efficient design with a D-error of 0.0282 is negligible.

So how long to you need to run the algorithm? For computationally simple syntax (e.g. MNL model with fixed priors) it only takes seconds or minutes for the D-error to stabilise to an acceptable level, while for computationally complex syntax (e.g. RPPANEL model with Bayesian priors) it may take hours or days. Many users simply run the syntax overnight and stop the algorithm the next morning.

Michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm

Re: Reference alternative in an orthogonal design

Postby Rim » Tue Jan 15, 2019 7:43 pm

Thank you very much for these answers, they are very helpful! :)
Rim
 
Posts: 6
Joined: Tue Dec 11, 2018 8:32 pm


Return to Choice experiments - general

Who is online

Users browsing this forum: No registered users and 5 guests

cron