Orthogonal or efficient?

[sharoncatherine]

Hello there,

I am a PhD student. As part of my project, I will conduct a DCE to examine the preferences of school leavers with respect to the attributes of third level institutions. I do not have any priors but I can make a educated guess with regard to the sign of the parameters.

?5 attributes
?Type; 2=university, 1=IoT, 0=College of education
?Travel time (from home); 1=1 hour, 2=2 hours, 3=3hours, 4=4 hours
?Course reputation; 2=excellent, 1=good, 0=fair
?Work placement; 1=offer work placement, 0=no work placement
?Fee; 1500, 3000, 4500, 6000

Initially, I planned to conduct an orthogonal design for my pilot study and then use the priors from this for the main survey. However, based on reading some of the posts from this forum, I now think that it might be better to use an efficient design. However, I have some questions if you don't mind.

1. Orthogonal design:
When I run an orthogonal design, Ngene says that it could not find a design based on 12 rows and defaults to 36. So, I have to create a design based on 36 rows and 3 blocks. This is the code that I used:

Design
; alts = alt1, alt2, alt3
; rows = 36
; orth = sim
; block = 3
; model:
u(alt1)= b1*type[2,1,0] + b2*travel_time[1,2,3,4] + b3*course_reputation[2,1,0] + b4.dummy*work_place[1,0] + b5*fee[1500,3000,4500,6000] /
u(alt2)= b1*type[2,1,0] + b2*travel_time[1,2,3,4] + b3*course_reputation[2,1,0] + b4.dummy*work_place[1,0] + b5*fee[1500,3000,4500,6000] $

Q: Do you think that this would be okay to do for my pilot study? I should have 30 respondents.

2: Efficient design:
Alternatively, I could run an efficient design. However, I have no priors so all I really have is the expected sign. See code below:

?Efficient design based on mnl model
?All priors set to +-0.001, depending on whether I expect the relationship to be + or -
?A single parameter is assigned to categorical attributes (course rep, type) - assuming for now that each category has the same impact on utility as I don't have proper priors.

Design
;alts = alt1*, alt2*, alt3
;rows = 12
;eff = (mnl,d)
;model:
U(alt1) = b1[0.001]*type[2,1,0] + b2[-0.001]*travel_time[1,2,3,4] + b3[0.001]*course_reputation[2,1,0] + b4.dummy[0.001]*work_place[1,0] + b5[-0.001]* fee[1500,3000,4500,6000] /
U(alt2) = b1*type + b2*travel_time + b3*course_reputation + b4.dummy*work_place + b5* fee $

Q: When I run this, the probability of choosing the opt-out (alt3) is over 90% in around 6/12 choice cards. Also, the s-estimate is enormous - although I think this is because the betas are so close to zero. Is the fact that the choice probability for alt3 is >90% an issue? Would I be better to do an orthogonal design?

I hope that my post is not too long but I would really appreciate any insights that you could give me.

Kind regards,
Sharon

[johnr]

Hi Sharon

Thanks for the post. Just to clarify, please note that an orthogonal design is an efficient design and any differentiation is actually artificial. For an MNL model with zero priors, the D-optimal design will most likely be an orthogonal design. Put another way, an D-efficient design generated for an MNL model with zero local priors will likely be an orthogonal design and hence, is by definition an efficient design. Rather than treat the design as an input as we have been doing, the literature should be treating the designs as outputs of the various assumptions analysts are making.
I'm simply pointing this out as I am on a mini crusade to ensure the literature is clear on this point.

To answer your specific question, there is no real right or wrong answer. The alternative that you might consider, and the one I would probably adopt is to use Bayesian priors, as shown below.

Design
;alts = alt1*, alt2*, alt3
;rows = 12
;eff = (mnl,d,mean)
;bdraws = halton(100)
;model:
U(alt1) = b1[(u,0,1)]*type[2,1,0] + b2[(u,-0.4,0)]*travel_time[1,2,3,4] + b3[(u,0,1)]*course_reputation[2,1,0] + b4.dummy[(u,0,2)]*work_place[1,0] + b5[(u,-0.00027,0)]* fee[1500,3000,4500,6000] /
U(alt2) = b1*type + b2*travel_time + b3*course_reputation + b4.dummy*work_place + b5* fee $

This optimises over the range shown. I have used uniform priors rather than Normals, as these represent weak priors (basically reflect direction only). To get the range, I basically calculated the midpoint of each attribute (for example, the midpoint of 1500 to 6000 is 3750. I then selected priors, such that the midpoint of the priors * the midpoint of X are the same for each attribute. That is for fee, the midpoint of -0.00027 and 0 is -0.00013, and -0.00013 * 3750 = -0.5 (or thereabouts). Likewise, the midpoint of 0 to 2 is 1, and the prior of 0 to 1 is 0.5, hence 1 * 0.5 = 0.5. By doing this, I have made the average contribution to utility to be roughly the same for each attribute (hence I am not saying one attribute is more or less important (on average) than any other. Whether this is the case empirically, I cannot say. Basically, you really want the real prior to appear in the range selected of the Bayesian prior distribution, but without additional information, I think the above is a safe strategy. If you look at your syntax based on local (fixed) priors, -0.001 * 3750 = -3.75 , and 0.001 * 1 = 0.001 , which suggests on average, fee is far far far more important in the design than type. I hope what I have written makes sense?

John