Not able to find a design

[agarwalmanoj]

Hello Dr. Bliemer,
I am trying to find a design for evaluating Uber type driver preferences. There are four attributes
• Wages per hour ($10, $20, $30)
• Minimum Weekly Hours (0,20,40)
• Benefits (Health & Retirement) (No, Healthcare only, Healthcare and Retirement)
• Passenger Rating by Drivers (No, Yes)

All variables are effects coded, and i want the main effect + interaction of Wages with the other three attributes. This is an unlabeled design.

I am having difficulty generating a design with the following code. I first tried the swap algorithm, but it does not even get to a design, I then switched to the modified Federov algorithm, and it is not able to find enough candidate sets.
"Error: The modified Federov candidate set size of 2000 could not be achieved. The percentages of candidates that failed are: 95.66% due dominance, 0% due constraints, and 4.34% due repeated alternatives. The candidate set size has been adjusted from 2000 to 1674."

Code
Design
; alts = alt1*,alt2*,none
; rows=9
; eff=(mnl,d)
;alg=mfederov
;model:
U(alt1) = b2.effects[-.18|-.12]*Wage[10,20,30] + b3.effects[.08 |.02]*Hours[0,20,40] + b4.effects[-.04|-.02]*BFT[0,1,2] +b5.effects[-.01]*RTNG[0,1]
+ int11[0.001]*Wage.effects[10]*Hours.effects[0] + int12[0.001]*Wage.effects[10]*Hours.effects[20] + int13[0.001]*Wage.effects[20]*Hours.effects[0] + int14[0.001]*Wage.effects[20]*Hours.effects[20]
+ int21[0.001]*Wage.effects[10]*BFT.effects[0] + int22[0.001]*Wage.effects[10]*BFT.effects[1] + int23[0.001]*Wage.effects[20]*BFT.effects[1] + int24[0.001]*Wage.effects[20]*BFT.effects[1]
+ int31[0.001]*Wage.effects[10]*RTNG.effects[0] + int32[0.001]*Wage.effects[20]*RTNG.effects[0] /

U(alt2) = b2*Wage +b3*Hours +b4*BFT +b5*RTNG
+ int11*Wage.effects[10]*Hours.effects[0] + int12*Wage.effects[10]*Hours.effects[20] + int13*Wage.effects[20]*Hours.effects[0] + int14*Wage.effects[20]*Hours.effects[20]
+ int21*Wage.effects[10]*BFT.effects[0] + int22*Wage.effects[10]*BFT.effects[1] + int23*Wage.effects[20]*BFT.effects[1] + int24*Wage.effects[20]*BFT.effects[1]
+ int31*Wage.effects[10]*RTNG.effects[0] + int32*Wage.effects[20]*RTNG.effects[0] /

U(none) = 0
$

Can you please help. I have just started to use NGENE, and I love its ease of use and provision of all relevant output in a convenient fashion.
Manoj Agarwal

[Michiel Bliemer]

Hi Manoj,

There were a few issues with the syntax, which I have fixed in the code below.

1. You cannot set a utility to zero. You either need to give it a constant, as I have in the code below, or you need to add a constant in alt1 and alt2 and remove the utility function for none since Ngene sets the utility by default to zero if it is not specified. The constant also needs a suitable prior, it is now defaulted to zero.

2. 9 rows is not enough for estimating so many interaction effects. I have applied my general rule of thumb by first computing the minimum number of rows needed, which is the number of estimated coefficients (17) divided by the number of alternatives minus one (2), multiplied by 2 or 3 to have sufficient variation in the data. To avoid giving to many choice tasks to a single respondent, I have blocked the design in 3.

3. The error message merely indicates that Ngene has adjusted the default candidate set size from 2000 to 1674. This is fine. You can avoid this message by setting the candidate set size as I have in the syntax below.

4. The main mistake you made was in specifying the same interaction effect twice, see below. I corrected this by changing the last 1 to 0. These mistakes are spotted more easily if you put the code in a more readable format as I have done below.
int23[0.001]*Wage.effects[20]*BFT.effects[1] + int24[0.001]*Wage.effects[20]*BFT.effects[1]

Code: Select all

Design
;alts = alt1*,alt2*,none
;rows=24
;block=3
;eff=(mnl,d)
;alg=mfederov(candidates = 1674)
;model:
U(alt1) = b2.effects[-.18|-.12] * Wage[10,20,30]
        + b3.effects[.08 |.02]  * Hours[0,20,40]
        + b4.effects[-.04|-.02] * BFT[0,1,2]
        + b5.effects[-.01]      * RTNG[0,1]
        + int11[0.001]          * Wage.effects[10] * Hours.effects[0]
        + int12[0.001]          * Wage.effects[10] * Hours.effects[20]
        + int13[0.001]          * Wage.effects[20] * Hours.effects[0]
        + int14[0.001]          * Wage.effects[20] * Hours.effects[20]
        + int21[0.001]          * Wage.effects[10] * BFT.effects[0]
        + int22[0.001]          * Wage.effects[10] * BFT.effects[1]
        + int23[0.001]          * Wage.effects[20] * BFT.effects[0]
        + int24[0.001]          * Wage.effects[20] * BFT.effects[1]
        + int31[0.001]          * Wage.effects[10] * RTNG.effects[0]
        + int32[0.001]          * Wage.effects[20] * RTNG.effects[0]
        /
U(alt2) = b2                    * Wage
        + b3                    * Hours
        + b4                    * BFT
        + b5                    * RTNG
        + int11                 * Wage.effects[10] * Hours.effects[0]
        + int12                 * Wage.effects[10] * Hours.effects[20]
        + int13                 * Wage.effects[20] * Hours.effects[0]
        + int14                 * Wage.effects[20] * Hours.effects[20]
        + int21                 * Wage.effects[10] * BFT.effects[0]
        + int22                 * Wage.effects[10] * BFT.effects[1]
        + int23                 * Wage.effects[20] * BFT.effects[0]
        + int24                 * Wage.effects[20] * BFT.effects[1]
        + int31                 * Wage.effects[10] * RTNG.effects[0]
        + int32                 * Wage.effects[20] * RTNG.effects[0]
        /
U(none) = b1[0]
$

Michiel

[agarwalmanoj]

Thanks so much for taking the time to fix my mistakes. Some follow up questions.

1. So I could also do the following in terms of specifying the utility functions and that would work
U(alt1) = bo [0] + b2*effects{-.18|-.12]*Wage[10,20,30) etc /
U(alt2) = b0 +b2 * Wage etc /
$
Any preference for which form is easier on the algorithm?

2 Should i use the modified federov or the swap algorithm for this problem. The reason is that i did not impose any level balance constraints yet in the modified federov, and would prefer the default swap algorithm, which will also try for level balance.

3a. Since there are 17 parameters, I calculated that 9 choice cards will be the minimum. What is your experience if we use the minimum? Can I get away with 12 choice cards? The reason is that for our model, we want to estimate individual level parameters, and want to avoid blocking across respondents.

3b. If we were to estimate only one aggregate model across respondents, could i get away with only 9 choice cards?

Thanks so much. I have learnt a lot from the previous questions and your answers. In your next edition of the manual, you may want to include some of the pointers you discuss in your Q&A in this forum. Thanks. Manoj

[Michiel Bliemer]

1. Yes, this specifcation is the same. There is no difference for the algorithm.

2. The default swapping algorithm will not be able to find a design since the dominance checks across alt1 and alt2 impose a large number of constraints that makes it difficult (impossible) for the swapping algorithm to find a design that satisfies all constraints. The modified Federov algorithm is more capable in finding designs with many (dominance and possibly other) constraints and is in this case the only option. There is no need to impose attribute level balance constraints in your case since all attributes are effects coded; for attributes that are dummy/effects coded, an efficient design will typically automatically have a high degree of attribute level balance since imbalance leads to inefficiency (i.e., information about dummy/effects coded coefficients is only captured when their associate levels appear within the design; having a high imbalance results in one or more dummy/effects coded coefficients getting little Fisher information and therefore increases the D-error). You can check the degree of attribute level balance in the design and I suspect that it is reasonably balanced. Note that you need to check for attribute level balance ACROSS alt1 and alt2 (since these are unlabelled alternatives) by looking at the 2*24 = 48 levels of each attribute.

3a. You will likely get away with 12 rows if you estimate an MNL model, although I generally would prefer more variation in the data (which also helps efficiency). If you plan to estimate a mixed logit model, which has more parameters, you need more variation in the data and 12 rows may not work. I am not sure what you mean with "individual level parameters". If you are planning to estimate individual-specific parameters using Bayesian estimation, then I can understand that you may want to give all respondents the same choice tasks (although this is not strictly necessary since coefficients will asymptotically converge to the same values independent of the choice tasks given to respondents). If you are planning to use maximum likelihood estimation, then perhaps you refer to individual-specific conditional parameter estimates (obtained as a post-processing step) but I do not see why using a larger design with blocking would be an issue.

3b. Theoretically yes, but finding a D-efficient design with 9 rows that can estimate all coefficients in the model will be challenging. You can try yourself by choosing ;rows = 9 and Ngene will likely be unable to locate a design that can estimate all interaction effects (which requires that each attribute level combination appears within the design) as Ngene will report that the design has an Undefined (infinite) D-error. Ngene is able to find a design with a finite D-error with 12 rows but still struggles. The many interaction effects in the model make it difficult to use a small design.

I appreciate your suggestion regarding the manual. We are working on an update of the manual which will indeed address the frequently asked questions here on the forum.

Michiel

[agarwalmanoj]

Thank you for your wonderful insights.
Manoj