choice-metrics.com

Dear Ngene team,

I am working on a design for a pretest with a status quo option and with 3 categorical attributes and one price attribute (4x3x3x6 levels). The status quo option does not vary across choice cards. It contains the base levels of the categorical attributes and a lowest price level, which is the only level not present in the other 2 alternatives.

I need about 8 choice cards, since K(9:16), depending on the model to be estimated, and generate a design with 18 draws and 2 blocks.
Since the number of levels is mixed, I have problems with level balance. For the last attribute - price, which is continuous and with 6 levels - the differences between the levels are small, so level balance gets less important. But for the first attribute, which is effects coded, level balance is a problem. However, I want to estimate parameters for the base levels too and would therefore prefer to use effects coding.

Code: Select all

Design
;alts = alt1*, alt2*, SQ
;rows = 18
;block = 2
;bdraws = gauss(2)
;eff = (mnl,d,mean)
;alg = mfederov(candidates=20)
;require:
SQ.TS=3,
SQ.AS=2,
SQ.US=2
;model:
U(alt1) = t.effects[(u,0,1)|(u,0,0.5)|(u,-0.5,0.5)] *  TS[0,1,2,3] + a.effects[(u,0,1)|(u,0,0.5)] * AS[0,1,2] +
u.effects[(u,0,1)|(u,0,0.5)] * US[0,1,2] +
p[(u,-1,0)] * P[0.7,0.8,0.9,1,1.1,1.2]/
U(alt2) = t * TS + a * AS + u * US + p * P/
U(SQ) = const[(u,0,1)] + t * TS + a * AS + u * US + p * Psq[0.6]$

I have several issues with the above design:
1. Ngene hangs up when I include more than 20 candidates for mfederov, finding a design is obviously difficult in this case. Is using 20 candidates ok?
2. Should I stay with the coding I used for the pretest also in the main survey?
3. I do not have much information on the priors, but some on the signs and the possible rankings of levels. Are the priors I specified general enough?
4. If I understand correctly, it is recommended to do a pretest with an MNL-optimized design. Does this also hold for the main survey design given the constraints included?
5. Is it appropriate to use 6 levels for the price attribute with the given narrow range? I have tried using 4 levels and 16 rows, but this leads to higher Derrors and more distorted level balance.
6. I would like to estimate interaction effects – with 18 rows I get for a model with one interaction effect Derror = 0.57 and for two interaction effects - Derror = 0.83. Do you recommend using the model with the two interaction effects, although it has greater Derror and the level balance is very bad? I tried imposing level balance constraints, but given the other model constraints Ngene is unable to find a design or hangs up. Is there another way to deal with the problem of having very bad level balance here?

I would appreciate any feedback from your side. Thanks very much in advance.
Nonka

Model with 1 interaction effect:

Code: Select all

?De=0.57, level balance is more or less ok, not good, but not too bad, S=67 and Sbayesian=289
Design
;alts = alt1*, alt2*, SQ
;rows = 18
;block = 2
;bdraws = gauss(2)
;eff = (mnl,d,mean)
;alg = mfederov(candidates=20)
;require:
SQ.TS=3,
SQ.AS=2,
SQ.US=2
;model:
U(alt1) = t.effects[(u,0,1)|(u,0,0.5)|(u,-0.1,0.1)] *  TS[0,1,2,3] + a.effects[(u,0,1)|(u,0,0.5)] * AS[0,1,2] +
u.effects[(u,0,1)|(u,0,0.5)] * US[0,1,2] +
p[(u,-1,0)] * P[0.7,0.8,0.9,1,1.1,1.2] +
int[(u,-1,1)]*TS.effects[1]*US.effects[0]/
U(alt2) = t * TS + a * AS + u * US + p * P  +
int[(u,-1,1)]*TS.effects[1]*US.effects[0]/
U(SQ) = const[(u,0,1)] + t * TS + a * AS + u * US + p * Psq[0.6]$

Model with 2 interaction effects:

Code: Select all

?De=0.83, level balance is bad for TS, S=78 and Sbayesian=383
Design
;alts = alt1*, alt2*, SQ
;rows = 18
;block = 2
;bdraws = gauss(2)
;eff = (mnl,d,mean)
;alg = mfederov(candidates=20)
;require:
SQ.TS=3,
SQ.AS=2,
SQ.US=2
;model:
U(alt1) = t.effects[(u,0,1)|(u,0,0.5)|(u,-0.1,0.1)] *  TS[0,1,2,3] + a.effects[(u,0,1)|(u,0,0.5)] * AS[0,1,2] +
u.effects[(u,0,1)|(u,0,0.5)] * US[0,1,2] +
p[(u,-1,0)] * P[0.7,0.8,0.9,1,1.1,1.2] +
int1[(u,-1,1)]*TS.effects[1]*US.effects[0] +
int2[(u,-1,1)]*TS.effects[2]*US.effects[0]/
U(alt2) = t * TS + a * AS + u * US + p * P  + int1*TS.effects[1]*US.effects[0] + int2*TS.effects[2]*US.effects[0]/
U(SQ) = const[(u,0,1)] + t * TS + a * AS + u * US + p * Psq[0.6]$

First of all, attribute level balance is not much of an issue, especially for dummy or effects coded variables. If an effects coded variable is not sufficiently attribute level balanced, then it simply becomes difficult to estimate the effects coded coefficients and hence the D-efficiency will be very poor. So a D-efficient design with dummy or effects coded variables automatically accounts for sufficient attribute level balance in the data. If you are still worried about it, you can tell Ngene to include a minimum number of each attribute, e.g. use TS[0,1,2,3](3-5,3-5,3-5,3-5]. Note that adds further (possibly unnecessary) constraints on the design.

Answers to your questions:

1. I could go up to 40 without problems, after that Ngene finds it difficult to find feasible choice tasks. It works fine without the status quo alternative, but somehow adding the status quo alternative makes it more difficult to find them. I would not expect this, so we will try to look into this further. I would not use 20 candidates, but if you can use 40 that should give you likely already sufficient flexibility.

2. You can change coding at a later stage, but your design was optimised for the coding specified and for the priors defined. Any deviation you have from these assumptions will lead to some efficiency in data collection. I would often not worry too much about loosing some efficiency, as your priors will never be exact anyway.

3. The sign of the price parameter is clear, but the ranking of the effects coded variables are not so clear. For effects (or dummy) coded variables, in order to rule out dominant alternatives (with the *) the ranking of the attribute levels is required. You do not specify a clear ranking, perhaps this is intentional, but if there does not exist any clear ranking then there is no need to worry about dominant alternatives and providing the sign does not matter much. If you think there is a clear ranking, then try to provide the ranking using your priors. Your priors are quite wide, which illustrates a high level of uncertainty about the values.

4. I am not sure I understand this question. What constraints do you mean? Your current design is an MNL optimised design taking the status quo constraints into account. I would optimise the design for the main study assuming an MNL model and using priors obtained from a pilot study (including their standard errors to inform Bayesian normally distributed priors). You can evaluate the design under a panel mixed logit model (but not optimise on it).

5. It is fine using 6 levels, it is usually more efficient to use less levels since this means that trade-offs are larger (trading off between 0.7 and 1.2 is more than between 1.1 and 1.2), but using 3 or 4 levels is usually a good compromise. I am not sure why your design would get less efficient with 4 levels, but maybe this is due to the large range in the prior distributions you are using.

6. I would add interaction effects if you intend to estimate them. I would not worry about attribute level balance, there is no reason to require attribute balance really, many people in marketing do not even impose attribute level constraints since they dummy or effects code everything. Attribute level balance is just another restriction that makes the design less efficient, so letting go of attribute level balance is good from an efficiency point of view. But perhaps you intend to estimate nonlinearities in price and then you need more balanced levels for price.

Hi Nonka

There are a few things going on in the code that seem to be preventing the design from being generated with more than about ~60 candidates. I think I have a solution, which involves a bit of a hack.

It seems that the only motivation for the Federov algorithm in your design is fixing the levels for the status quo alternative. Normally, this can be done by associating the generic parameter with an attribute that has a different name (e.g. TS_SQ) that has a single level only. This creates some problems though with dummy or effects coding. As an alternative to the Federov algorithm with a require property, how about the following:

Code: Select all

Design
;alts = alt1*, alt2*, SQ
;rows = 18
;block = 2
;bdraws = gauss(2)
;eff = (mnl,d,mean)
?;alg = mfederov(candidates=100)
;model:
U(alt1) =
t.effects[(u,0,1)|(u,0,0.5)|(u,-0.5,0.5)] * TS[0,1,2,3] +
a.effects[(u,0,1)|(u,0,0.5)]              * AS[0,1,2] +
u.effects[(u,0,1)|(u,0,0.5)]              * US[0,1,2] +
p[(u,-1,0)]                               * P[0.7,0.8,0.9,1,1.1,1.2] /
U(alt2) =
t * TS + a * AS + u * US + p * P/
U(SQ) =
const[(u,0,1)] + t * TSSQ[3.000001,3.000002,3.000003,3.000004] + a * ASSQ[2.000001,2.000002,2.000003] + u * USSQ[2.000001,2.000002,2.000003] + p * Psq[0.6]
$

The status quo alternative uses different attribute names, so that different levels can be specified. Ngene will require the same number of levels as the other attributes associated with the generic parameters (e.g. 4 for TS). It will also generate an error if the exact same levels are specified. So in the above code, I have specified levels that differ by a very small amount. This will have a negligible effect on efficiency, but allows a fairly level balanced design to be generated with the default swapping algorithm. When I ran this, there were many different designs evaluated.

Andrew

Dear Andrew,

thank you very much for your helpful comments and suggestions.

I have tried to decrease the uncertainty in the prior levels and to insert some sort of ranking, as much as possible. In addition I will have to insert a no-choice alternative (forth alternative), since leaving it out would make the experiment unrealistic. Nevertheless I would like to keep the levels of the third alternative fixed, as before. Ngene gives me some designs with up to 60 candidates for the mfederov. However I get better level balance and same D-error with 50 candidates. So I thought, I could stay with the 50-candidates design. Do you think it is OK?

I am not sure I can use the code you suggested - with 100 candidates and different levels for the fixed alternative, since I would like to estimate parameters for effects coded variables. In this respect, does it matter much whether 100 or 50 candidates have been used for the Federov algorithm? If not, then maybe I can use the design with the 50 candidates and require property.

I have also modified one of the interaction effects. I am actually interested in the interaction of one of the levels of the effects coded attribute US with the base level of the effects coded attribute TS. Ngene gave me such designs based on the code below. But, is it appropriate to include the base level of an effects coded attribute in an interaction term?

I would really appreciate your comments on these issues.
Thanks a lot in advance.

Best wishes,
Nonka

Code: Select all

? D-error = 0.522
Design
;alts = alt1*, alt2*, SQ, no
;rows = 16
;block = 2
;bdraws= gauss(2)
;eff = (mnl,d)
;alg = mfederov(candidates=50)
;require:
SQ.TS=3,
SQ.AS=2,
SQ.US=2
;model:
U(alt1) = t.effects[(u,0.022,0.05)|(u,0.011,0.021)|(u,-0.01,0.01)] *  TS[0,1,2,3] + a.effects[(u,0.006,0.01)|(u,0,0.005)] * AS[0,1,2] + u.effects[(u,0.011,0.021)|(u,0,0.01)] * US[0,1,2] + p[-0.04] * P[0.75,0.9,1.05,1.2] + int1[(n,0,0.01)]*TS.effects[1]*US.effects[0] + int2[(n,0,0.01)]*TS.effects[3]*US.effects[0]/
U(alt2) = t * TS + a * AS + u * US + p * P + int1*TS.effects[1]*US.effects[0] + int2*TS.effects[3]*US.effects[0]/
U(SQ) = const[(u,-0.01,0)] + t * TS + a * AS + u * US + p * Psq[0.6]$

1) If you include a no-choice alternative, you also need to include a constant and set an appropriate prior to get choice probabilities that make sense.

2) The number of candidates used does not effect the number of parameters you need to estimate, only the number of rows is relevant for that. Using a larger candidate set just means that Ngene has a larger number of rows it can select from, therefore offering a higher likelihood of finding a more efficient design.

3) Yes you can interact with the base level of an effects coded variable. Note that in effects coding, the base is set to [-1,-1,...,-1], not to zeros, hence there is no problem with such an interaction.

Michiel

Thank you very much for your quick reply.
And thank you for having this forum. It is very valuable and helpful.

Dear Andrew,
Dear Michiel,

sorry for bothering again. I included a constant for the no choice alternative, but now I have encountered another problem with the design. I always get one scenario with the same levels of the effects coded attributes in alt2 as in the fixed alternative only with higher price. I try to exclude this scenario by specifying another two constraints using "if", however on running the code ngene shows the error message: "The ';require' property contains an attribute that does not exist in the design. 'if(alt1.as'." I do have the attribute AS in alt1 and therefore I wonder why I get the error message.

Here is the code:

Code: Select all

Design
;alts = alt1*, alt2*, SQ, no
;rows = 16
;block = 2
;bdraws= gauss(2)
;eff = (mnl,d)
;alg = mfederov(candidates=100)
;require:
SQ.TS=3,
SQ.AS=2,
SQ.US=2,
if(alt1.AS=2 and alt1.US=2, alt1.TS<3),
if(alt2.AS=2 and alt2.US=2, alt2.TS<3)
;model:
U(alt1) = t.effects[(u,0.022,0.05)|(u,0.011,0.021)|(u,-0.01,0.01)] *  TS[0,1,2,3] + a.effects[(u,0.006,0.01)|(u,0,0.005)] * AS[0,1,2] + u.effects[(u,0.011,0.021)|(u,0,0.01)] * US[0,1,2] + p[-0.04] * P[0.78,0.96,1.14,1.32] + int1[(n,0,0.01)]*TS.effects[1]*US.effects[0] + int2[(n,0,0.01)]*TS.effects[3]*US.effects[0]/
U(alt2) = t * TS + a * AS + u * US + p * P + int1*TS.effects[1]*US.effects[0] + int2*TS.effects[3]*US.effects[0]/
U(SQ) = sqcst[(u,-0.01,0)] + t * TS + a * AS + u * US + p * Psq[0.6]/
U(no)= nocst[-0.001]$

Best regards,
Nonka

You are mixing constraints that cannot be mixed.
You can only use "if" constraints in the ;cond (conditions) command, not in the ;require command. The "if" constraints use the swapping algorithm, the ;require and ;reject command use the modified Federov command. You can only use ;cond OR ;require/;reject, but not both.

Thanks again!

Hallo again,

I now have the results of an mnl estimation based on a pilot study with 50 respondents. I got insignificant estimates for some of the effects coded parameters, which do not give me a clear ranking of the parameters.

Moreover, depending on whether I include the interaction effects in the model or not, the ranking of two of the levels for the 4-level attribute (TS) changes. But in both cases the estimates are insignificant. For one of the levels of the 3rd attribute (US) I get quite different, but again insignificant results, depending on whether I estimate the interactions. This level is the level which is included in both interaction effects.

Another problem I have is that I get very high s-estimates for the Bayesian design with interaction effects, without interaction effects the s-estimates are more reasonable. By fixing most of the priors, I can reduce the S-estimates for the design with interaction effects.

However, since there is still high uncertainty for some of the priors, and I am conducting an online survey, I think I could do an additional test study. I could get some 100 more responses either using the initial choice cards from the pilot or the cards I generate with Bayesian priors or fixed priors from the pilot. What strategy do you think I should choose?

Very many thanks in advance and all the best for 2017!
Best wishes,
Nonka