choice-metrics.com

Hello.
I am trying to create a choice experiment with 5 attributes in which we try to value the WTP for different elements of a renewable energy product.
The attributes have the following levels: distance (4), size (3), carbon reduced (4), jobs created (4), fee added monthly to electricity bill (8).
I have run 2 focus groups to obtain priors for this design, but the designs I get in Ngene don't seem to be in line with my intuition of a good design.
The syntax I have been using is the following:

Code: Select all

Design
;alts = alt1,alt2,sq
;rows=24
;block=3
;eff=(mnl,wtp(ref1))
;wtp = ref1(*/b6)
;model:
U(alt1) = b2[(n,0.099014,0.021367)]*dist[5,8,12,18] + b3[(n,-0.00279,0.001203)]*size[64,100,144]+ b4[(n,0.000000321,0.000000484)]*cars[200000, 300000, 400000, 500000] + b5[(n,0.00035,0.00029)]*jobs[500, 800, 1100, 1400]+ b6[(n,-0.06408,0.021047)]*fee[0,3,6,9,14,20,25,30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee/
U(sq) = sq[0.05]*sq[1]
$

My status quo has the following characteristics: distance: 35, size: 0, cars: 200 000, jobs: 500 and fee: 0

When I run the above, I get many scenarios with dominated choice alternatives or alternatives which are exactly the same OR alternatives which vary only in one attribute but have such a big fee differential that they create dominated alternatives. Also, Ngene always pairs the highest fee with the closest (worst) distance, and the lower fees with the furthest (best) distances. Also, the highest fee is always paired with the MOST carbon reduction, while the lowest fee is almost always paired with the HIGHEST carbon reduction. This doesn't really provide enough variation without causing correlation between these attribute levels....

Is there something I am doing wrong?
Does anyone have a suggestion how to better model this problem?
Any tips would be much appreciated.
Kind regards.

I have tried to simplify the problem by taking the uncertainty of the priors out as well as reduce dominance by starring the alternatives as in the following code:

Code: Select all

Design
;alts(m1)= alt1*,alt2*,sq
;rows=24
;block=3
;eff= M1(mnl,d)
;model(m1):
U(alt1) = b2[0.099]*dist[5,8,12,18] + b3[-0.0028]*size[64, 100,144]+ b4[0.0028]*cars[200000, 300000,400000, 500000] + b5[0.00032]*jobs[500,800,1100,1400]+ b6[-0.064]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee/
U(sq) = sq[0.005]
$

This does, however, not seem to work either - as Ngene doesn't seem to find a design to meet the criteria. What I suspect the problem is is that the priors are too uninformative.
It also seems like I might be doing something to specify the signs wrong (although I cannot see what this might be) as when I specify a simple orthogonal design and inflate the priors (as by the code below), I still get dominated alternatives.

Code: Select all

Design
;alts= alt1*,alt2*
;rows=24
;orth=seq
;block=3
;model:
U(alt1) = b2[0.5]*dist[5,8,12,18] + b3[-0.1]*size[64, 100,144]+ b4[0.075]*cars[200000, 300000,400000, 500000] + b5[0.75]*jobs[500,800,1100,1400]+ b6[-0.4]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee

The only way that I have been able to obtain a design which doesn't offer dominated alternatives is to specify priors for each of the effects-coded levels (I didn't specify the attributes as effects coded at the beginning because priors were already rather uninformative when specified linearly) and in a sense "make up" priors with the same ordinal nature as the priors, but much larger in absolute value:

Code: Select all

Design
;alts= alt1*,alt2*
;rows=24
;block=3
;eff =(mnl,d)
;model:
U(alt1) = b2.effects[-0.6|-0.45|-0.2]*dist[5,8,12,18] + b3.effects[-0.1|-0.4]*size[100,144,64]+ b4.effects[0.08|0.12|0.16]*cars[300000,400000, 500000,200000] + b5.effects[0.08|0.12|0.16]*jobs[800,1100,1400,500]+ b6[-0.5]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee
$

Is there any way to justify doing the above? Surely it is scientifically frowned upon to create my own priors, but I can't seem to find a different way to specify the design in Ngene in order to produce a good design.
Any advice would be much appreciated.

The actual priors from the focus group are the following if run with effects codes (sorry, I couldn't get this to display in a easily legible manner) :

Code: Select all

                   Coefficient   Std. Error          Z
distance   5   -0.6781673   0.2167049         -3.13
          8   -0.1290687    0.166465         -0.78
         12   0.2210336            0.1644681   1.34
size        100   -0.0803065    0.0815758   -0.98
       144   -0.0388423    0.0690564   -0.56
jobs      800   0.0436941            0.1048728   0.42
     1100   -0.4077483      0.158552   -2.57
          1400        -0.0274241   0.1552703           -0.18
carbon   300   0.2012015     0.1179523    1.71
          400   -0.094157     0.1294262           -0.73
          500   0.1434007            0.1106058   1.3
price              -0.0608513   0.0202454           -3.01
sq              -0.4776386   0.4898253            -0.98

and the following if run in a linear model (note: intuitively the sign on jobs coefficient is wrong).

Code: Select all

        Coefficient   Std. Error       Z
distance   0.0990142     0.0213674       4.63
size           -0.0027851   0.0012027       -2.32
jobs           -0.0003485   0.0002897        -1.2
carbon   3.21E-07       4.84E-07       0.66
price    -0.064084      0.021047      -3.04
sq             -2.894862   0.568178       -5.09

I would like to suggest to specify the status quo alternative in a different way, as follows:

Code: Select all

Design
;alts = alt1*,alt2*,sq
;rows=24
;block=3
;eff=(mnl,d)
;model:
U(alt1) = b2[0.099014]*dist[5,8,12,18] + b3[-0.00279]*size[64,100,144]+ b4[0.000000321]*cars[200000, 300000, 400000, 500000] + b5[0.00035]*jobs[500, 800, 1100, 1400]+ b6[-0.06408]*fee[0,3,6,9,14,20,25,30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee/
U(sq) = b1[0] + b2*distsq[35] + b3*sizesq[0] + b4*carsq[200000] + b5*jobsq[500] + b6*feesq[0]
$

Notice that i put the the levels of the status quo alternative directly into the utility function. You have to specify the prior for the constant (b1), which may not be zero as I have specified.

Using the stars in alt1* and alt2* should rule out any strictly dominant alternatives (i.e., alternatives that are better in all attributes). However, it does not prevent from weakly dominant alternatives to appear, i.e., alternatives that are chosen with a high probability. If I run this syntax, it appears that the status quo alternative is chosen with a probability of more than 90%, making it weakly dominant, and therefore not generating much information. is the status quo alternative indeed too much preferred? Are the priors correct?

Michiel

Hi Michiel.

Thank you so much for that quick reply.
I was trying what you mentioned, but I also wanted to incorporate the effects-coded nature that the final model I use will take - and I noticed that doesn't work if I specify the status quo the way you showed me. The other issue with our status quo is that it uses the following levels: distance(35), size (0), jobs(200), cars(100000) and fee(0)... none of which are found in the levels for the other alternatives.

I found a model which produces a pretty reasonable design (with good tradeoffs as far as I can tell) and I just wanted to check with you if it makes sense what I'm doing. I ran the following model averaging approach to try and emphasize balance between the alternatives (John Rose did a similar model before which I tried to adapt here).

Code: Select all

Design
;alts(m1)= alt1,alt2,sq
;alts(m2)= alt1, alt2
;rows=24
;block=3
;eff= M1(mnl,d)+ 0.01*M2(mnl,b)
;model(m1):
U(alt1) = b2.effects[(n,-0.94329,0.194845)|(n,-0.04167,0.156817)|(n,0.347576,0.139605)]*dist[5,8,12,18] + b3.effects[(n,-0.0084,0.063759)|(n,-0.12212,0.061054)]*size[100,144,64]+ b4[(n,0.000000443,0.000000501)]*cars[200000, 300000,400000, 500000] + b5[(n,0.00025,0.000325)]*jobs[500,800,1100,1400]+ b6[(n,-0.07041,0.021274)]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee/
U(sq) = sq[(n,-0.68138,0.484523)]

;model(m2):
U(alt1) = b2[(n,0.099014,0.021367)]*dist[5,8,12,18] + b3[(n,-0.00279,0.001203)]*size[100,144,64]+ b4[(n,0.000000321,0.000000484)]*cars[200000,300000,400000, 500000] + b5[(n,0.0003485,0.0002897)]*jobs[500,800,1100,1400]+ b6[(n,-0.06408,0.021047)]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee
$

I obtained the priors from the model "m1" by running a mixed model. Although the final model will probably include effects codes for cars (carbon reduced) and jobs, I specify these attributes as a linear here, just because the priors were incredibly uninformative even if I assume the variables to be continuous (adding effects coding just robbed the model and variables of more significance).
The second lot of priors are obtained by simply running a linear model.
Is this design acceptable?
Thank you very much for your advice.

I would say, if the choice tasks make sense to you, then the design is acceptable. There is not really just one right way of doing it, so I guess your approach would work.

Just being a bit creative, see the syntax below. I added the appropriate levels of the status quo to the attributes, and then constrain the design to force the levels of the status quo to be set to a certain level. Perhaps this will give you some more ideas.

Code: Select all

Design
;alts= alt1*,alt2*,sq
;rows=24
;block=3
;eff= (mnl,d)
;alg = mfederov(candidates = 500)
;require:
sq.dist = 35,
sq.size = 0,
alt1.dist < 35,
alt1.size > 0,
alt2.dist < 35
;model:
U(alt1) = b2.effects[0|-0.94329|-0.04167|0.347576]*dist[35,5,8,12,18] + b3.effects[0|-0.0084|-0.12212]*size[0,100,144,64]+ b4[0.000000443]*cars[200000, 300000,400000, 500000] + b5[0.00025]*jobs[500,800,1100,1400]+ b6[-0.07041]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee /
U(sq) = b2*dist + b3*size +b4*carssq[100000] +b5*jobssq[200] +b6*feesq[0]
$

Hi Michiel.
Looks like a great design - would love to try it, but the program crashes every time I try to run it. I've tried on 2 computers now - no success.
Any tips?
Thanks again!

You are running the exact same syntax, or did you make any changes?
It runs on my computer... I am assuming you have the latest version of Ngene?

Yes, new version and exactly the same syntax.
Tried it on two computers (which are both very good computers), but it doesn't seem to run without crashing the program....
I did get it to run once yesterday... but since then not again.
I tried to adjust the code to specify
alt2.size>0
also, because otherwise I get results which don't make sense given the nature of the problem...

Since adding constraints make it harder to find a design, try decreasing the number of candidates in the Modified Federov algorithm, for example set it to 100. It should run. As it is a random generation of candidates, sometimes it will find them more easily than other times.
Adding alt2.size > 0 does not work, as then no base levels appear in any other alternative than the status quo, which really makes the status quo alternative the base (and the model does not estimate, the D-error is undefined).
If you cannot have the base alternative to appear in the other alternatives, then I guess your earlier suggestion by just putting a constant in the utility function of the status quo is an option, although not ideal.

Hello Michiel.

You're right - the design does run with setting the number of candidates to 100.
The problem in our case is indeed that the status quo alternative is the base (all the levels of the status quo do not vary with the alternatives presented in the choices).... while this may not be ideal, it best mirrors the problem at hand. We are planning to estimate the coefficients of all the levels (except the status quo level) by using the hybrid coding structure used by Cooper, Rose and Crase (2011 - Does anybody like water restrictions?).

Given this, the design below gives the choice alternatives that make the most sense of all designs I have tried.

Would you be able to look over it one more time and see if there is something that doesn't make sense, something I missed, or whether something is odd in my design? We hope to print our surveys this week and unfortunately none of my colleagues know Ngene and don't have much experience with designing choice experiments. The choice alternatives make sense to our eyes - but as mentioned, they're far from expert eyes! I would greatly appreciate the eyes of an expert to spot any obvious problem.

Thanks and have a great week!

Code: Select all

Design
;alts(m1)= alt1,alt2,sq
;alts(m2)= alt1, alt2
;rows=24
;block=3
;eff= M1(mnl,d)+ 0.01*M2(mnl,b)

;model(m1):
U(alt1) = b2.effects[(n,-0.94329,0.194845)|(n,-0.04167,0.156817)|(n,0.347576,0.139605)]*dist[5,8,12,18] + b3.effects[(n,-0.0084,0.063759)|(n,-0.12212,0.061054)]*size[100,144,64]+ b4[(n,0.000000443,0.000000501)]*cars[200000, 300000,400000, 500000] + b5[(n,0.00025,0.000325)]*jobs[500,800,1100,1400]+ b6[(n,-0.07041,0.021274)]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee/
U(sq) = sq[(n,-0.68138,0.484523)]

;model(m2):
U(alt1) = b2[(n,0.099014,0.021367)]*dist[5,8,12,18] + b3[(n,-0.00279,0.001203)]*size[100,144,64]+ b4[(n,0.000000321,0.000000484)]*cars[200000,300000,400000, 500000] + b5[(n,0.0003485,0.0002897)]*jobs[500,800,1100,1400]+ b6[(n,-0.06408,0.021047)]*fee[2, 5, 8, 12, 16, 20, 25, 30] /
U(alt2) = b2*dist + b3*size +b4*cars +b5*jobs +b6*fee
$