How reliable is a code giving a single design?

[Arnaud Blaser]

Dear Ngene's programers/masters/gods,

We are currently working on implementing a Choice experiment which aims at being very simple; It is a "new heating system" choice task with three alternatives having four attributes (energy savings in %; 10, 20, 30, 40, 50, 60 / net costs savings in monetary terms; 200, 400, 600, 800, 1000, 1200 / uncertainty regarding costs saving in %; 10, 20, 30, 40, 50, 60 / and some dummies for technology).

Basically the idea is that net costs savings are different from energy savings, for instance because the new system has higher maintenance costs than the previous one. Further there is an uncertainty level regarding costs savings, which might come from fuel price variation or unreliable new technology.

As a starting point I tried to generate a design with the following code;

Code: Select all

Design
;alts = alt1*, alt2*, alt3*
;rows = 12
;block = 3
;

;eff=(mnl,d)
;model:
U(alt1) = btech.dummy[0|0|0] * tech[0,1,2,3] + beff[0.01] * eff[10,20,30,40,50,60]
 + bsave[0.001] * save[200,400,600,800,1000,1200] + bvar[-0.02] * var[10,20,30,40,50,60]/
U(alt2) = btech * tech + beff * eff + bsave * save + bvar * var /
U(alt3) = btech * tech + beff * eff + bsave * save + bvar * var $

This gives a design which looks reasonable, and has covariance matrix and choice probabilities which also looks reasonable to me;

NL covariance matrix
btech(d0) btech(d1) btech(d2) beff bsave bvar
1.018421 0.443631 0.746482 0.00547 -0.000137 -0.003238
0.443631 0.902768 0.488037 0.005556 7.6E-05 -0.005548
0.746482 0.488037 1.535192 0.012572 -0.000394 -0.004437
0.00547 0.005556 0.012572 0.000801 1.6E-05 -0.000429
-0.000137 7.6E-05 -0.000394 1.6E-05 2E-06 -1.8E-05
-0.003238 -0.005548 -0.004437 -0.000429 -1.8E-05 0.000666



alt1 alt2 alt3
0.40296 0.29852 0.29852
0.182703 0.449378 0.367919
0.436752 0.239694 0.323554
0.332225 0.367165 0.30061
0.355913 0.322043 0.322043
0.41642 0.376792 0.206788
0.340003 0.152773 0.507224
0.26639 0.439203 0.294407
0.309344 0.461488 0.229168
0.463963 0.281408 0.254629
0.167537 0.276221 0.556242
0.333333 0.333333 0.333333


I have one big concern however; this code leads to a single design (rather than the usual many evaluations). I tried to move a little bit the priors (we basically know only the sign) but it does not change much (sometimes design fails entierly with slightly different priors). If I try the same code with two alternatives ten it run smoothly. My question is; how reliable can it be and how to fix that? The problem clearly comes from too much dominated alternatives being generated.

My main question is; how reliable can be a code which found a single? Is there any remote chance that this design is worth something?

My secondary questions are obviously on how to proceed to fix this;

My hypothesis is that in order to deal with this we need to stick to two alternatives (if we want to keep the attributes as they are) or we need to add more attributes in order to help Ngene to find undominated alternatives. Is this correct?

I think that adding levels to the current attributes is not really an option since it is a 12 rows design and all attributes already have 6 levels, hence the only way to keep balance while increasing the number of levels would be to have 12 levels attributes. Is this right, or are 12 levels attributes potentially helpful here?

ps: sorry for the ugly tables above, is there a way to insert/format them to be more readable on the forum?

[Michiel Bliemer]

I think your requirement of strict balance together with non-dominancy is quite difficult. I would suggest to relax the restriction of attribute level balance somewhat. How about this syntax?

Code: Select all

Design
;alts = alt1*, alt2*, alt3*
;rows = 12
;block = 3
;alg = mfederov(candidates = 1000)
;eff = (mnl,d)
;model:
U(alt1) = btech.dummy[0|0|0] * tech[0,1,2,3](2-4,2-4,2-4,2-4) + beff[0.01] * eff[10,20,30,40,50,60](1-3,1-3,1-3,1-3,1-3,1-3)
+ bsave[0.001] * save[200,400,600,800,1000,1200](1-3,1-3,1-3,1-3,1-3,1-3) + bvar[-0.02] * var[10,20,30,40,50,60](1-3,1-3,1-3,1-3,1-3,1-3)/
U(alt2) = btech * tech + beff * eff + bsave * save + bvar * var /
U(alt3) = btech * tech + beff * eff + bsave * save + bvar * var $

Note that since you block it in 3, you will not have attribute level balance anyway.

[Arnaud Blaser]

Thanks for the suggestion. I will discuss how badly we need attributes balance with the team.

Providing we decide that we really require 3 alternatives with strict balance, which approach (in your experience) is the most likely to help Ngene; adding more attributes or more levels?

[Michiel Bliemer]

I would not suggest adding more levels, as this usually makes the design less efficient (fewer but extreme levels are usually more efficient), but changing the number of attributes is an option. Do you really need 3 alternatives? You can easily find designs with two alternatives that are attribute level balanced and that do not contain dominant alternatives, as generated using the syntax below.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 12
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = btech.dummy[0|0|0] * tech[0,1,2,3] + beff[0.01] * eff[10,20,30,40,50,60]
+ bsave[0.001] * save[200,400,600,800,1000,1200] + bvar[-0.02] * var[10,20,30,40,50,60]/
U(alt2) = btech * tech + beff * eff + bsave * save + bvar * var $

Adding more alternatives will likely lead to several alternatives being dominated in a certain choice task.

[Arnaud Blaser]

Thanks for the answer. Yes the team's decision was to include three alternatives instead of two to make the choice more realistic. I think we will go a design based on the federov algo; we realised that we'll need to add further constrain on some attributes' level display hence balance will not be achievable anyway.