When doing "orth=ood", how much D-optimality is acceptable?

[Lohas]

Dear Michiel, I am going to do an "orthorgonal" design, due to my limited project time.

I am designing the DCE with 6 attributes, including price, (2 with 4 levels, 2 with 3 levels, 2 with 2 levels, the price is the attribute with 4 levels). I wrote the syntax as below:
when "orth=seq":
Design
;alts = alt1*,alt2*
;rows = 72
;orth = seq
;block = 12
;model:
U(alt1) = b1*A[1,2,3,4] + b2*B[1,2,3] + b3*C[1,2,3] + b4*D[1,2] + b5*E[1,2] + b6*F[45,50,55,60]/
U(alt2) = b1*A + b2*B + b3*C + b4*D + b5*E + b6*F
$

when "orth=ood":
Design
;alts = alt1*,alt2*
;rows = 72
;orth = ood
;block = 12
;model:
U(alt1) = b1*A[1,2,3,4] + b2*B[1,2,3] + b3*C[1,2,3] + b4*D[1,2] + b5*E[1,2] + b6*F[45,50,55,60]/ ?F[45,50,55,60] is th price attribute
U(alt2) = b1*A + b2*B + b3*C + b4*D + b5*E + b6*F
$

When I run the design of "orth=seq", I got the D-optimality around 60%-69%, D error is around 0.029688.
When I run the design of "orth=ood", I got the D optimality fixed at 96.25861, D error is fixed as 0.017916.

So my questions are:
1) Is everything about my syntax right? As an unlabelled design, I am going to put a "No choice" alternative in the questionnaire too. so do I need to add "alt3" here in the syntax, or the syntax with "alt2" and "alt3"is ok?
2) "orth=seq", or "orth=ood", which one should I use?
When I use the OOD design, I found that the attribute with 2 levels are always shown in the choice set as level1 vs. level2, it always ask the respondent to make trade-off between level1 and level2, not
showing any choice set as level1 vs. level1.
3) How many levels is better for Price attribute? 3 or 4? How much interval is ok for the price, so that I won't get an result for price attribute which is not significant after I collected all the data?

Many thanks! Looking forward to your reply!

Kind regards,
Lohas

[Michiel Bliemer]

1) All looks fine. The design will not change when you add the no choice (alt3) alternative, but the D-error will change, so it is good practice to add the no choice alternative with a constant, i.e. U(alt3) = b0. I notice that you use levels 1,2,3 except for F. It is highly unusual to use such levels when estimating a model, perhaps you want to use dummy coding instead. This will affect your D-error since this would mean estimating more parameters.

2) Both are fine. OOD designs are a special type of SEQ design where trade-offs are maximised. It will therefore always force respondents to make a trade-off, i.e. level 1 versus level 2 and never level 1 versus level 1 since this is less efficient. I do not use orthogonal designs myself, for pilot studies I typically use efficient designs with zero priors.

3) Both 3 or 4 levels would be fine. I am unable to advice regarding range or sample size since this is different for each study. It is best to do a pilot study, estimate parameters to use as priors and generate a (Bayesian) efficient design from which you can also obtain a sample size estimate.

Michiel

[Lohas]

1) All looks fine. The design will not change when you add the no choice (alt3) alternative, but the D-error will change, so it is good practice to add the no choice alternative with a constant, i.e. U(alt3) = b0. I notice that you use levels 1,2,3 except for F. It is highly unusual to use such levels when estimating a model, perhaps you want to use dummy coding instead. This will affect your D-error since this would mean estimating more parameters.

2) Both are fine. OOD designs are a special type of SEQ design where trade-offs are maximised. It will therefore always force respondents to make a trade-off, i.e. level 1 versus level 2 and never level 1 versus level 1 since this is less efficient. I do not use orthogonal designs myself, for pilot studies I typically use efficient designs with zero priors.

3) Both 3 or 4 levels would be fine. I am unable to advice regarding range or sample size since this is different for each study. It is best to do a pilot study, estimate parameters to use as priors and generate a (Bayesian) efficient design from which you can also obtain a sample size estimate.

Michiel

Hi Michiel, thanks very much for your reply. following some further Qs.

1) For the attribute F, I used price directly, and I found that the D-error in this case is 0.017916, however, if I coded the levels in F as [1,2,3,4], the D-error raised to 0.030637. SO, how much D-optimality or D-error is acceptable?

2) I choose the orthogonal design because I have very limited time before the due time of my project. So, if I do a pilot study use efficient design, how many survey respondents do I need to get at least?

3) I have a question about the "attribute selection", what if the researcher's interest on testing some attribute of a product is not within the list of attributes that get from the consumer's focus group? (e.g. I want to test a beef steak that selling on e-commerce platform, I am interested to see the consumer's preference on different e-commerce channels, such as purchase online and deliver to consumer's home, or purchase online and consumer collect the product at the offline shop of the e-commerce channel. But this channel attribute doesn't show out from the consumer group result.)

4) I have a Q about the number of the attributes. Currently I have 6 attributes: A,B,C,D,E,F. So, if the result show that the attribute E and F are statistical significant, but A,B,C,D are not. But I cares more about the attributes A,B,C,D, I want to make these 4 attribute statistical significant. In this case, if I exclude the attribute E and F, only test attributes A,B,C,D in the DCE, will the result of statistical significance change?

Many thanks Michiel, and looking forward to your reply.

Best regards,
Lohas

[Michiel Bliemer]

1) It is not possible to interpret the D-error since it is case specific, sometimes a value of 0.1 is good, other times a value of 0.1 is bad. If you are creating an orthogonal design and use levels 1,2,3,4 you can ignore the D-error as it is meaningless. You do not need 100% D-optimality, you can still estimate the model if you have a D-optimality of 10% but you just need a larger sample size.

2) You can choose to create a D-efficient design with zero priors instead of an orthogonal design, both would work and take the same amount of time to create. At least test your design by collecting some data from a few respondents (e.g. 2 or 3 of your colleagues) and use appropriate coding (e.g. dummy/effects coding) to ensure that you can estimate your model. In order to get informative priors for creating an efficient design with non-zero priors, you would need more respondents, but how much is case specific (sometimes 5 respondents is enough, other times you need more than 50).

3) You can add any attribute you like and test its impact on choice (but it may turn out statistically not significant).

4) The result may or may not change. If respondents mainly focus on E and F and ignore the others then excluding E and F will make respondents focus on A,B,C,D and the result may change. However, if A,B,C,D are deemed irrelevant by the respondents then the results may not change. If you would like respondents to make trade-offs only between A,B,C,D then you could consider adding attributes E and F as scenario variables, i.e. their levels are constant across alternatives 1 and 2.

Michiel

[Lohas]

1) It is not possible to interpret the D-error since it is case specific, sometimes a value of 0.1 is good, other times a value of 0.1 is bad. If you are creating an orthogonal design and use levels 1,2,3,4 you can ignore the D-error as it is meaningless. You do not need 100% D-optimality, you can still estimate the model if you have a D-optimality of 10% but you just need a larger sample size.

2) You can choose to create a D-efficient design with zero priors instead of an orthogonal design, both would work and take the same amount of time to create. At least test your design by collecting some data from a few respondents (e.g. 2 or 3 of your colleagues) and use appropriate coding (e.g. dummy/effects coding) to ensure that you can estimate your model. In order to get informative priors for creating an efficient design with non-zero priors, you would need more respondents, but how much is case specific (sometimes 5 respondents is enough, other times you need more than 50).

3) You can add any attribute you like and test its impact on choice (but it may turn out statistically not significant).

4) The result may or may not change. If respondents mainly focus on E and F and ignore the others then excluding E and F will make respondents focus on A,B,C,D and the result may change. However, if A,B,C,D are deemed irrelevant by the respondents then the results may not change. If you would like respondents to make trade-offs only between A,B,C,D then you could consider adding attributes E and F as scenario variables, i.e. their levels are constant across alternatives 1 and 2.

Michiel

Dear Michiel,
Many thanks for your help! I am a novice of DCE, I attended the 2018 training, but this is my first time to apply the DCE. So many basic questions. Hope that won't bother you.

Thanks again.
Lohas

[Lohas]

1) It is not possible to interpret the D-error since it is case specific, sometimes a value of 0.1 is good, other times a value of 0.1 is bad. If you are creating an orthogonal design and use levels 1,2,3,4 you can ignore the D-error as it is meaningless. You do not need 100% D-optimality, you can still estimate the model if you have a D-optimality of 10% but you just need a larger sample size.

2) You can choose to create a D-efficient design with zero priors instead of an orthogonal design, both would work and take the same amount of time to create. At least test your design by collecting some data from a few respondents (e.g. 2 or 3 of your colleagues) and use appropriate coding (e.g. dummy/effects coding) to ensure that you can estimate your model. In order to get informative priors for creating an efficient design with non-zero priors, you would need more respondents, but how much is case specific (sometimes 5 respondents is enough, other times you need more than 50).

3) You can add any attribute you like and test its impact on choice (but it may turn out statistically not significant).

4) The result may or may not change. If respondents mainly focus on E and F and ignore the others then excluding E and F will make respondents focus on A,B,C,D and the result may change. However, if A,B,C,D are deemed irrelevant by the respondents then the results may not change. If you would like respondents to make trade-offs only between A,B,C,D then you could consider adding attributes E and F as scenario variables, i.e. their levels are constant across alternatives 1 and 2.

Michiel

Dear Michiel,

I have question about the attribute combination issues. As I got 2 attribute to test, one is "traceability" with 2 levels (Traceability information provided by QR code, and None); the other attribute is "Country of Origin", with 2 levels (Country of origin logo, and None).
My question is:
1) Can these 2 attributes be combined into 1 attribute with 3 levels (Traceability information provided by QR code, Country of Origin logo, and None)?
2) What's the difference of the results if I use 2 separate attributes, or 1 combined attribute?
Kind regards,
Lohas

[Michiel Bliemer]

These are not Ngene-related questions so should be posted in the other forum, but a quick reply:

1. Yes you can do that.
2. Separating them allows 4 levels instead of 3:

... b1.dummy[0.1] * qrcode[1,0] + b2.dummy[0.2] * country[1,0] + ...

This means:
* a utility of 0 if there is no qr code and no country of origin
* a utility of 0.1 if there is a qr code and no country of origin
* a utility of 0.2 if there is a country of origin and no qr code
* a utility of 0.3 if there is a qr code and a country of origin

Otherwise, you get:

... b1.dummy[0.1|0.2] * traceability[1,2,0] ...

This means:
* a utility of 0 if there is no qr code and no country of origin
* a utility of 0.1 if there is a qr code and no country of origin
* a utility of 0.2 if there is a country of origin and no qr code

Michiel

[Lohas]

These are not Ngene-related questions so should be posted in the other forum, but a quick reply:

1. Yes you can do that.
2. Separating them allows 4 levels instead of 3:

... b1.dummy[0.1] * qrcode[1,0] + b2.dummy[0.2] * country[1,0] + ...

This means:
* a utility of 0 if there is no qr code and no country of origin
* a utility of 0.1 if there is a qr code and no country of origin
* a utility of 0.2 if there is a country of origin and no qr code
* a utility of 0.3 if there is a qr code and a country of origin

Otherwise, you get:

... b1.dummy[0.1|0.2] * traceability[1,2,0] ...

This means:
* a utility of 0 if there is no qr code and no country of origin
* a utility of 0.1 if there is a qr code and no country of origin
* a utility of 0.2 if there is a country of origin and no qr code

Michiel

Many thanks, Michiel! I will post the Design-related questionnaires to the other forum.
Best regards,
Lohas

[Lohas]

1) All looks fine. The design will not change when you add the no choice (alt3) alternative, but the D-error will change, so it is good practice to add the no choice alternative with a constant, i.e. U(alt3) = b0. I notice that you use levels 1,2,3 except for F. It is highly unusual to use such levels when estimating a model, perhaps you want to use dummy coding instead. This will affect your D-error since this would mean estimating more parameters.

2) Both are fine. OOD designs are a special type of SEQ design where trade-offs are maximised. It will therefore always force respondents to make a trade-off, i.e. level 1 versus level 2 and never level 1 versus level 1 since this is less efficient. I do not use orthogonal designs myself, for pilot studies I typically use efficient designs with zero priors.

3) Both 3 or 4 levels would be fine. I am unable to advice regarding range or sample size since this is different for each study. It is best to do a pilot study, estimate parameters to use as priors and generate a (Bayesian) efficient design from which you can also obtain a sample size estimate.

Michiel

Dear Michiel,

1. Regarding your reply in 1): Is it better to code the "Price" attribute as: ...+ b6*F[1,2,3,4], instead of ...+ b6*F[45,50,55,60] in a OOD design?
2. I limited my attributes number to 4 in total (3 product attribute, 1 price attribute) at the end, and just wonder if 4 attributes for a DCE will be fine and it it won't be too less for a design right?

As above, these are my final question about Ngene by far I think. Many thanks in advance!

Best regards,
Lohas

[Michiel Bliemer]

1. OOD designs do not care about the attribute levels, the design will be the same.
2. Ngene can handle any number of attributes, it is up to the analyst to determine the number of relevant attributes.

Michiel