Beginner: review a design and some questions

[hblee]

Dear Moderators,

I am happy to join this forum of Ngene. I am trying to ensure I am going in the right direction. Let me describe:

Three alternatives

Alt1: product (unlabeled)
Alt2: product (unlabeled)
Alt3: neither (I would not choose option A and B)

Three attributes

Att1: a generic attribute measured by percentage, two levels, [20%, 30%]
Att2: a generic discrete attribute, five levels, [0, 1, 2, 3, 4]
Att3: price, three levels, [$20, $30, $40]

Research purposes and models

Att2 is about production practices and has five levels. Among the five levels, we have an interest in two levels especially. The two levels represent two different environmentally friendly practices (two eco-labels).

The main purpose is to estimate the marginal WTP for the two eco-labels. Specifically,

(1) Estimate the WTP for the two eco-labels, including means and more information about the WTP distribution
(2) Examine how WTP for the two eco-labels varies with some respondent characteristics (for example, income, gender, age)
(3) Predict potential market shares of eco-labeled products

For (1), I plan to use a random parameter logit (i.e., mixed logit) to examine the WTP distribution, considering heterogeneous preference.
For (2), I plan to consider interaction terms between eco-label dummies and some covariates.

Procedures

Step 1. For the pilot study,
Generate efficient design with zero priors.

Code: Select all

Design
;alts = alt1*, alt2*, alt3 ? alt1 and alt2 are unlabeled products

;rows = 60 ? 60 is needed for balanced

;eff = (mnl,d) ? Use a D-efficiency design because it is most widely used

;alg = swap(stop=total(10000 iterations)) ? Criterion for stopping evaluations

;block = 4 ? Each respondent will answer 15 choice tasks (=60/4)

;model:
U(alt1) = b11 + b2 * A[0.20,0.30] + b3.dummy[0|0|0|0] * B[0,1,2,3,4] + b4 * C[20,30,40] /
U(alt2) = b12 + b2 * A + b3 * B + b4 * C           
$


Step 2: Run pilot, estimate for MNL.

Step 3. Actual survey

Use priors and SE's from the pilot to design a d-efficient design.

Use the same code as the pilot except for priors.


A request

Is the procedure reasonable in general? Do I miss something? Any feedback would be very appreciated. For information, I put some comments next to each code sentence.

Questions

Question 1. Above I use ;eff = (mnl,d) for both the pilot and the actual survey, although I plan to use a MMNL (mixed logit). My decision is based on the finding of Bliemer and Rose (2010): the efficient design for the MNL model performed relatively efficient for estimating panel mixed logit models, although some efficiency will be lost. Is it a good approach?

Question 2. I put the criterion for stopping evaluations because I found evaluations took very long in some cases. I thought 10,000 iterations would be sufficient. Is it a good approach?

Question 3. I plan to consider some interaction effects between eco-labels and some economic characteristics of respondents. Respondent characteristics of interest are continuous. But I do not consider the interactions in designing experiments. Is it a good approach?

Any feedback would be very appreciated. Thank you.

[Michiel Bliemer]

That looks like a great plan.

Q1. Yes I would always recommend optimising a design for an mnl model, which will also be quite efficient for estimating a panel mixed logit model. Optimising for rppanel is too computationally intensive for practical use is most cases.

Q2. I generally impose no stopping criterion and just look at how fast the D-error goes down. In this case, Ngene finds a sufficiently efficient design after a few minutes and can no longer improve, so you could manually stop the search probably after 10 minutes or so. For Bayesian efficient designs (for your main survey) I would simply run the search overnight.

Q3. At the design stage you only include attributes, including possible interactions effects between attributes, but you would generally not include socio-economic variables at the design stage, only in the estimation stage. Only if you expect vastly different responses between respondents you could create a different design for different segments using something like gender.covar in your script, but otherwise you would not include them now.

Two comments on your script:

1. When using zero priors, I would recommend dummy coding ALL attributes, including numerical attributes. The reason is that with (near) zero priors it becomes optimal to make comparisons only using the most extreme attribute levels. If you look in your design, alt1.price vs alt2.price is mostly (20,40) and (30,30), where level 30 is "sacrificed" to mostly allow extreme comparisons 20-40. If you use dummy coding then attribute level combinations appear more equally.

2. Is there no preference order for attribute 2? If not, then I guess there is no real risk for alternative dominance. If all attributes have a preference order, then please specify something like b3.dummy[-0.001|-0.002] * price[30,40,20] to indicate that price=30 is preferred less relative to price 20, and price=40 is preferred less than price=30, noting that the last level in Ngene (in this case, 20) is the reference level.

Michiel

[hblee]

Thank you for your kind response!

Q1. Thank you for your explanation.

Q2. Thank you for your explanation. I will follow your practice.

Q3. Thank you for your explanation.

Thank you for the additional two comments.

About your two comments, I found that the two comments were related, so let me discuss the two together.

Based on #1 comment, I coded att1 and att2 as dummy attributes. But I keep using zero-priors on att1 and att2. Then, designs were generated. But still, price attribute likely has extreme comparisons.

Based on #2 comment, now I put non-zero priors, in addition to dummy coding. Then, an error message appeared, and Ngene failed to find a design. I copy and paste the trace box:

Warning: Defaulting to prior values of zero for the following priors: 'b11, b12'
Note: Defaulting to assigning blocks with the 'minsum' method.

A valid initial random design could not be generated after approximately 10 seconds. In this time, of the 209199 attempts made, there were 0 row repetitions, 9855 alternative repetitions, and 199344 cases of dominance. There are a number of possible causes for this, including the specification of too many constraints, not having enough attributes or attribute levels for the number of rows required, and the use of too many scenario attributes. A design may yet be found, and the search will continue for 10 minutes. Alternatively, you can stop the run and alter the syntax.

For information, the code using non-zero priors is as follows:

Code: Select all

Design
;alts = alt1*, alt2*, alt3 ? alt1 and alt2 are unlabeled products

;rows = 60 ? 60 is needed for balanced

;eff = (mnl,d) ? Use a D-efficiency design because it is most widely used

? Do not use a stop criterion. See the d-error and stop manually.

;block = 4 ? Each respondent will answer 15 choice tasks (=60/4)

;model:
U(alt1) = b11 + b2.dummy[0.001] * A[0.30,0.20] + b3.dummy[0|0|0|0] * B[0,1,2,3,4] + b4.dummy[-0.001|-0.002] * C[30,40,20] /
U(alt2) = b12 + b2 * A + b3 * B + b4 * C           
$

? dummy coding for all attributes
? put non-zero priors on dummy variables


For information, I used several different non-zero priors for exploration, but the same error message appeared.

My new approach: Sequential orthogonal fractional factorial design

Now I am thinking of using a sequential orthogonal fractional factorial design. I have three reasons.

Reason 1. Orthogonal fractional factorial designs are widely used in the literature.

Reason 2. A "sequential" orthogonal design is needed when using unlabeled alternatives for orthogonal designs.

Reason 3. Price attribute levels appeared in a balanced way. That is, all price pairs appeared, and the frequency was similar across pairs.

Specifically, I used the following code:

Code: Select all

Design
;alts = alt1*, alt2*, alt3 ? alt1 and alt2 are unlabeled products

;rows = 60 ? 60 is needed for balanced

;orth = seq ? seq orth is needed not to make repeated choice tasks for orthogonal designs

;block = 4 ? Each respondent will answer 15 choice tasks (=60/4)

;model:
U(alt1) = b11 + b2.dummy[0] * A[0.30,0.20] + b3.dummy[0|0|0|0] * B[0,1,2,3,4] + b4.dummy[-0.001|-0.002] * C[30,40,20] /
U(alt2) = b12 + b2 * A + b3 * B + b4 * C           
$


Questions

Question A. Do you have a guess why the error message appeared? Do you have any suggestions to solve the error?

Question B. Now I am happy with the sequential orthogonal fractional factorial design. So, I wonder about your general opinion on my new approach. Do I have something to consider?

Any additional feedback would be greatly appreciated.

Thank you again for your help.

[Michiel Bliemer]

QA: Yes, the warning message tells you that there are "199344 cases of dominance". In other words, Ngene could not find 60 choice tasks without any dominant alternatives using the default swapping algorithm. The solution is to add ;alg = mfederov to the script so that it switches to the modified Federov algorithm, which is a row-based algorithm. Efficient designs can automatically avoid dominant alternatives.

QB: You can use the sequential orthogonal design, but it will contain dominant alternatives. As long as you understand the impact of dominant alternatives in your experiment and account for them in your analysis it is fine. It is not possible to avoid dominant alternatives when using an orthogonal design since orthogonality is a very strong constraint.

See also: https://www.sciencedirect.com/science/article/pii/S0191261517304228

Michiel

[hblee]

Thank you very much for your kind response. I am sorry for the delayed response. I planned to reply after I was ready to ask related questions, but it has taken more time than I expected. I will get back to your last response when I am ready.

Thank you again.