unlabelled experiment, some attributes depend on others

[YangHU]

Code: Select all

Code

design
;alts = opt1*, opt2*, opt3*
;block = 1
? efficient design
;eff = (mnl, d) 
;rows = 27 
;cond:
if(opt1.contract = 1, opt1.plug_in_hours = [90,120,150]),
if(opt1.contract = 1, opt1.rewards1 = [50,110,170]),
if(opt1.contract = 1, opt1.rewards2 = 0),
if(opt1.contract = 1, opt1.rewards3 = 0), 
if(opt1.contract = 2, opt1.plug_in_hours = 0),
if(opt1.contract = 2, opt1.rewards1 = 0),
if(opt1.contract = 2, opt1.rewards2 = [0.7,1.0,1.3]),
if(opt1.contract = 2, opt1.rewards3 = 0), 
if(opt1.contract = 3, opt1.plug_in_hours = 0),
if(opt1.contract = 3, opt1.rewards1 = 0),
if(opt1.contract = 3, opt1.rewards2 = 0),
if(opt1.contract = 3, opt1.rewards3 = [0.1,0.15,0.2]),       
if(opt2.contract = 1, opt2.plug_in_hours = [90,120,150]),
if(opt2.contract = 1, opt2.rewards1 = [50,110,170]),
if(opt2.contract = 1, opt2.rewards2 = 0),
if(opt2.contract = 1, opt2.rewards3 = 0), 
if(opt2.contract = 2, opt2.plug_in_hours = 0),
if(opt2.contract = 2, opt2.rewards1 = 0),
if(opt2.contract = 2, opt2.rewards2 = [0.7,1.0,1.3]),
if(opt2.contract = 2, opt2.rewards3 = 0), 
if(opt2.contract = 3, opt2.plug_in_hours = 0),
if(opt2.contract = 3, opt2.rewards1 = 0),
if(opt2.contract = 3, opt2.rewards2 = 0),
if(opt2.contract = 3, opt2.rewards3 = [0.1,0.15,0.2]),       
if(opt3.contract = 1, opt3.plug_in_hours = [90,120,150]),
if(opt3.contract = 1, opt3.rewards1 = [50,110,170]),
if(opt3.contract = 1, opt3.rewards2 = 0),
if(opt3.contract = 1, opt3.rewards3 = 0), 
if(opt3.contract = 2, opt3.plug_in_hours = 0),
if(opt3.contract = 2, opt3.rewards1 = 0),
if(opt3.contract = 2, opt3.rewards2 = [0.7,1.0,1.3]),
if(opt3.contract = 2, opt3.rewards3 = 0), 
if(opt3.contract = 3, opt3.plug_in_hours = 0),
if(opt3.contract = 3, opt3.rewards1 = 0),
if(opt3.contract = 3, opt3.rewards2 = 0),
if(opt3.contract = 3, opt3.rewards3 = [0.1,0.15,0.2])     

;model:
U(opt1) = b1[0.00000000000000001 ]         * SOC[50,100,150]         + b2[0]         * plug_in_hours[90,120,150,0]         + b3[0]         * rewards1[50,110,170,0]         + b4[0]         * rewards2[0.7,1.0,1.3,0]         + b5[0]         * rewards3[0.1,0.15,0.2,0]         + b6.dummy[0|0] * contract[1,2,3] /
U(opt2) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract /
U(opt3) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract
$



Error message
Error message :

Note: Defaulting to assigning blocks with the 'minsum' method. Note: The conditional statement nesting cluster 1 contains 15 permissible combinations of attribute levels. The nesting cluster contains the following if statements: * if(opt1.contract = 1, opt1.plug_in_hours = [90,120,150]) * if(opt1.contract = 1, opt1.rewards1 = [50,110,170]) * if(opt1.contract = 1, opt1.rewards2 = 0) * if(opt1.contract = 1, opt1.rewards3 = 0) * if(opt1.contract = 2, opt1.plug_in_hours = 0) * if(opt1.contract = 2, opt1.rewards1 = 0) * if(opt1.contract = 2, opt1.rewards2 = [0.7,1.0,1.3]) * if(opt1.contract = 2, opt1.rewards3 = 0) * if(opt1.contract = 3, opt1.plug_in_hours = 0) * if(opt1.contract = 3, opt1.rewards1 = 0) * if(opt1.contract = 3, opt1.rewards2 = 0) * if(opt1.contract = 3, opt1.rewards3 = [0.1,0.15,0.2]) The conditional statement nesting cluster 2 contains 15 permissible combinations of attribute levels. The nesting cluster contains the following if statements: * if(opt2.contract = 1, opt2.plug_in_hours = [90,120,150]) * if(opt2.contract = 1, opt2.rewards1 = [50,110,170]) * if(opt2.contract = 1, opt2.rewards2 = 0) * if(opt2.contract = 1, opt2.rewards3 = 0) * if(opt2.contract = 2, opt2.plug_in_hours = 0) * if(opt2.contract = 2, opt2.rewards1 = 0) * if(opt2.contract = 2, opt2.rewards2 = [0.7,1.0,1.3]) * if(opt2.contract = 2, opt2.rewards3 = 0) * if(opt2.contract = 3, opt2.plug_in_hours = 0) * if(opt2.contract = 3, opt2.rewards1 = 0) * if(opt2.contract = 3, opt2.rewards2 = 0) * if(opt2.contract = 3, opt2.rewards3 = [0.1,0.15,0.2]) The conditional statement nesting cluster 3 contains 15 permissible combinations of attribute levels. The nesting cluster contains the following if statements: * if(opt3.contract = 1, opt3.plug_in_hours = [90,120,150]) * if(opt3.contract = 1, opt3.rewards1 = [50,110,170]) * if(opt3.contract = 1, opt3.rewards2 = 0) * if(opt3.contract = 1, opt3.rewards3 = 0) * if(opt3.contract = 2, opt3.plug_in_hours = 0) * if(opt3.contract = 2, opt3.rewards1 = 0) * if(opt3.contract = 2, opt3.rewards2 = [0.7,1.0,1.3]) * if(opt3.contract = 2, opt3.rewards3 = 0) * if(opt3.contract = 3, opt3.plug_in_hours = 0) * if(opt3.contract = 3, opt3.rewards1 = 0) * if(opt3.contract = 3, opt3.rewards2 = 0) * if(opt3.contract = 3, opt3.rewards3 = [0.1,0.15,0.2]) An attempt will be made to balance the frequency of each level in attributes affected by constraints, however complete balance might not be possible. A valid initial random design could not be generated after approximately 10 seconds. In this time, of the 26899 attempts made, there were 0 row repetitions, 1969 alternative repetitions, and 24930 cases of dominance. There are a number of possible causes for this, including the specification of too many constraints, not having enough attributes or attribute levels for the number of rows required, and the use of too many scenario attributes. A design may yet be found, and the search will continue for 10 minutes. Alternatively, you can stop the run and alter the syntax.



My questions:

If all prior is set as “0”, the Ngene could provide the results. If some values are set here, the error message will occur and it will not provide any results. the constraints are not especially strong in our case (it's a quite standard unlabelled experiment, in which some attributes depend on other attributes).

For the time being, the report is telling us the it allows for 15 combinations for alternative one, 15 for alternative 2, and 15 for 3, which is manageable. It also mention that is struggling with attribute balance (which we do not necessarily need; so it should be lifted...I guess. So how could I lift the attributes balance? Or how could proceed ? Many thanks.

[Michiel Bliemer]

The issue is with dominance, Ngene checks all attributes with non-zero priors for dominance. It is not possible to check for dominance when only one attribute has a non-zero prior to indicate the preference order of the attribute. You should either set all priors to zero, or you should provide information about other attributes as well. For example, in the script below I provided some priors for all attributes except contract (but I am just guessing here). The script will still not run because the default swapping algorithm will struggle to find 27 choice tasks without any dominance. This can be resolved by switching to the modified Federov algorithm. This algorithm does not allow ;cond constraints, but allows ;reject constraints. I rewrite your constraints into reject constraints, see script below. This script runs and will remove choice tasks with dominant alternatives. The modified Federov algorithm will not satisfy attribute level balance, so you may see that the middle levels are not used much for the numerical attributes. To resolve this, you could add attribute level constraints, e.g. (5-10,5-10,5-10,5-10) to indicate that each of the 4 levels should appear a minimum of 5 times and a maximum of 10 times. Alternatively, you can dummy code all attributes (which is fine when you have near-zero priors) such that all attribute levels will more or less appear equally automatically. Of course you can still estimate the model later without dummy coding.

Code: Select all

design
;alts = opt1*, opt2*, opt3*
;block = 1
? efficient design
;eff = (mnl, d)
;rows = 27
;alg = mfederov
;reject:
opt1.contract = 1 and opt1.plug_in_hours = 0,
opt1.contract = 1 and opt1.rewards1 = 0,
opt1.contract = 1 and opt1.rewards2 > 0,
opt1.contract = 1 and opt1.rewards3 > 0,
opt1.contract = 2 and opt1.plug_in_hours > 0,
opt1.contract = 2 and opt1.rewards1 > 0,
opt1.contract = 2 and opt1.rewards2 = 0,
opt1.contract = 2 and opt1.rewards3 > 0,
opt1.contract = 3 and opt1.plug_in_hours > 0,
opt1.contract = 3 and opt1.rewards1 > 0,
opt1.contract = 3 and opt1.rewards2 > 0,
opt1.contract = 3 and opt1.rewards3 = 0,

opt2.contract = 1 and opt2.plug_in_hours = 0,
opt2.contract = 1 and opt2.rewards1 = 0,
opt2.contract = 1 and opt2.rewards2 > 0,
opt2.contract = 1 and opt2.rewards3 > 0,
opt2.contract = 2 and opt2.plug_in_hours > 0,
opt2.contract = 2 and opt2.rewards1 > 0,
opt2.contract = 2 and opt2.rewards2 = 0,
opt2.contract = 2 and opt2.rewards3 > 0,
opt2.contract = 3 and opt2.plug_in_hours > 0,
opt2.contract = 3 and opt2.rewards1 > 0,
opt2.contract = 3 and opt2.rewards2 > 0,
opt2.contract = 3 and opt2.rewards3 = 0,

opt3.contract = 1 and opt3.plug_in_hours = 0,
opt3.contract = 1 and opt3.rewards1 = 0,
opt3.contract = 1 and opt3.rewards2 > 0,
opt3.contract = 1 and opt3.rewards3 > 0,
opt3.contract = 2 and opt3.plug_in_hours > 0,
opt3.contract = 2 and opt3.rewards1 > 0,
opt3.contract = 2 and opt3.rewards2 = 0,
opt3.contract = 2 and opt3.rewards3 > 0,
opt3.contract = 3 and opt3.plug_in_hours > 0,
opt3.contract = 3 and opt3.rewards1 > 0,
opt3.contract = 3 and opt3.rewards2 > 0,
opt3.contract = 3 and opt3.rewards3 = 0

;model:
U(opt1) = b1[ 0.00001]        * SOC[50,100,150]         
        + b2[-0.00001]        * plug_in_hours[90,120,150,0]         
        + b3[ 0.00001]        * rewards1[50,110,170,0]         
        + b4[ 0.00001]        * rewards2[0.7,1.0,1.3,0]         
        + b5[ 0.00001]        * rewards3[0.1,0.15,0.2,0]         
        + b6.dummy[0|0]       * contract[1,2,3]
        /
U(opt2) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract /
U(opt3) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract
$

Michiel

[YangHU]

Dear Michiel,

Thanks a lot! It actually works. This issue here:

The modified Federov algorithm will not satisfy attribute level balance, so you may see that the middle levels are not used much for the numerical attributes. To resolve this, you could add attribute level constraints, e.g. (5-10,5-10,5-10,5-10) to indicate that each of the 4 levels should appear a minimum of 5 times and a maximum of 10 times.

Could you please indicate how to code attribute level constraints ? Attched is the updated code. As I am a starter of N-gene.Many thanks.

Code: Select all

design
;alts = opt1*, opt2*, opt3*
;block = 4
? efficient design
;eff = (mnl, d, mean)
;rows = 24
;alg = mfederov
;bdraws = sobol(200)

;reject:
opt1.contract = 1 and opt1.plug_in_hours = 0,
opt1.contract = 1 and opt1.rewards1 = 0,
opt1.contract = 1 and opt1.rewards2 > 0,
opt1.contract = 1 and opt1.rewards3 > 0,
opt1.contract = 2 and opt1.plug_in_hours > 0,
opt1.contract = 2 and opt1.rewards1 > 0,
opt1.contract = 2 and opt1.rewards2 = 0,
opt1.contract = 2 and opt1.rewards3 > 0,
opt1.contract = 3 and opt1.plug_in_hours > 0,
opt1.contract = 3 and opt1.rewards1 > 0,
opt1.contract = 3 and opt1.rewards2 > 0,
opt1.contract = 3 and opt1.rewards3 = 0,

opt2.contract = 1 and opt2.plug_in_hours = 0,
opt2.contract = 1 and opt2.rewards1 = 0,
opt2.contract = 1 and opt2.rewards2 > 0,
opt2.contract = 1 and opt2.rewards3 > 0,
opt2.contract = 2 and opt2.plug_in_hours > 0,
opt2.contract = 2 and opt2.rewards1 > 0,
opt2.contract = 2 and opt2.rewards2 = 0,
opt2.contract = 2 and opt2.rewards3 > 0,
opt2.contract = 3 and opt2.plug_in_hours > 0,
opt2.contract = 3 and opt2.rewards1 > 0,
opt2.contract = 3 and opt2.rewards2 > 0,
opt2.contract = 3 and opt2.rewards3 = 0,

opt3.contract = 1 and opt3.plug_in_hours = 0,
opt3.contract = 1 and opt3.rewards1 = 0,
opt3.contract = 1 and opt3.rewards2 > 0,
opt3.contract = 1 and opt3.rewards3 > 0,
opt3.contract = 2 and opt3.plug_in_hours > 0,
opt3.contract = 2 and opt3.rewards1 > 0,
opt3.contract = 2 and opt3.rewards2 = 0,
opt3.contract = 2 and opt3.rewards3 > 0,
opt3.contract = 3 and opt3.plug_in_hours > 0,
opt3.contract = 3 and opt3.rewards1 > 0,
opt3.contract = 3 and opt3.rewards2 > 0,
opt3.contract = 3 and opt3.rewards3 = 0






;model:
U(opt1) = b1[(n,0.0174,0.00239)]        * SOC[50,100,150]         
        + b2[(n,-0.0202,0.00661)]        * plug_in_hours[90,120,150,0]         
        + b3[(n,0.0118,0.00325)]        * rewards1[50,110,170,0]         
        + b4[(n,1.2,0.593)]          * rewards2[0.7,1.0,1.3,0]         
        + b5[(n,5.38,2.98)]        * rewards3[0.1,0.15,0.2,0]         
        + b6.dummy[0.107343|-0.668332]       * contract[1,2,3]
        /
U(opt2) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract /
U(opt3) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract
$


Yang

[Michiel Bliemer]

The constraints you are imposing are quite strict, namely if contract = 1 than rewards2 = 0 and rewards3 = 0, so there are not many cases rewards are larger than zero and hence achieving attribute level balance is challenging. Also, plug_in_hours = 0 except for contract = 1, so in order to estimate the parameter for this attribute it tries to have more options with contract = 1 than contract = 2 or 3.

Perhaps most important is to impose constraints on how often each level of contract appears, so between 6 and 10 times (perfect balance would be 6 times for each level across 24 rows).

In the script below I imposed level constraints on contract, and I also included (imbalance) in the efficiency measure such that it tries to minimise 2*D-error + 1*imbalance, you can find the equation for imbalance in the Ngene manual. Also, I increased the number of rows in the candidate set to 5000 (default is 2000) to allow more flexibility for Ngene to select choice tasks in the design. I suggest you give this script a try and make further improvements if needed. You may want to run it for a while (at least a few hours) since the modified Federov algorithm is quite slow, especially with Bayesian priors. Note that any constraint you impose on the levels reduces the efficiency of the design.

Code: Select all

design
;alts = opt1*, opt2*, opt3*
;block = 4
? efficient design
;eff = 2*(mnl, d, mean) + (imbalance)
;rows = 24
;alg = mfederov(candidates = 5000)
;bdraws = sobol(200)

;reject:
opt1.contract = 1 and opt1.plug_in_hours = 0,
opt1.contract = 1 and opt1.rewards1 = 0,
opt1.contract = 1 and opt1.rewards2 > 0,
opt1.contract = 1 and opt1.rewards3 > 0,
opt1.contract = 2 and opt1.plug_in_hours > 0,
opt1.contract = 2 and opt1.rewards1 > 0,
opt1.contract = 2 and opt1.rewards2 = 0,
opt1.contract = 2 and opt1.rewards3 > 0,
opt1.contract = 3 and opt1.plug_in_hours > 0,
opt1.contract = 3 and opt1.rewards1 > 0,
opt1.contract = 3 and opt1.rewards2 > 0,
opt1.contract = 3 and opt1.rewards3 = 0,

opt2.contract = 1 and opt2.plug_in_hours = 0,
opt2.contract = 1 and opt2.rewards1 = 0,
opt2.contract = 1 and opt2.rewards2 > 0,
opt2.contract = 1 and opt2.rewards3 > 0,
opt2.contract = 2 and opt2.plug_in_hours > 0,
opt2.contract = 2 and opt2.rewards1 > 0,
opt2.contract = 2 and opt2.rewards2 = 0,
opt2.contract = 2 and opt2.rewards3 > 0,
opt2.contract = 3 and opt2.plug_in_hours > 0,
opt2.contract = 3 and opt2.rewards1 > 0,
opt2.contract = 3 and opt2.rewards2 > 0,
opt2.contract = 3 and opt2.rewards3 = 0,

opt3.contract = 1 and opt3.plug_in_hours = 0,
opt3.contract = 1 and opt3.rewards1 = 0,
opt3.contract = 1 and opt3.rewards2 > 0,
opt3.contract = 1 and opt3.rewards3 > 0,
opt3.contract = 2 and opt3.plug_in_hours > 0,
opt3.contract = 2 and opt3.rewards1 > 0,
opt3.contract = 2 and opt3.rewards2 = 0,
opt3.contract = 2 and opt3.rewards3 > 0,
opt3.contract = 3 and opt3.plug_in_hours > 0,
opt3.contract = 3 and opt3.rewards1 > 0,
opt3.contract = 3 and opt3.rewards2 > 0,
opt3.contract = 3 and opt3.rewards3 = 0


;model:
U(opt1) = b1[(n,0.0174,0.00239)]        * SOC[50,100,150] 
        + b2[(n,-0.0202,0.00661)]        * plug_in_hours[90,120,150,0]     
        + b3[(n,0.0118,0.00325)]        * rewards1[50,110,170,0]       
        + b4[(n,1.2,0.593)]          * rewards2[0.7,1.0,1.3,0]       
        + b5[(n,5.38,2.98)]        * rewards3[0.1,0.15,0.2,0]         
        + b6.dummy[0.107343|-0.668332]       * contract[1,2,3](6-10,6-10,6-10)
        /
U(opt2) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract /
U(opt3) = b1 * SOC         + b2 * plug_in_hours         + b3 * rewards1         + b4 * rewards2         + b5 * rewards3         + b6 * contract
$

Michiel

[YangHU]

Dear Michel,

Thanks for your help. It actually works. However, the results also show more appearence contract = 1, compared with contract = 2 or 3, while we set the 6-10 times for the apperance of each contract.

My only concern is that respondents have strong preferences for contract = 2 and contract = 3 (based on the pilot survey). Shall I impose constraints with contract = 1 for 6 times apperance, as i am a little worried more appearance of contract= 1 will not make them compares with different types of attributes in contract = 1 rather than just selecting the third one.

For instance(see following choice set), respodents who have a strong preference for contract 3 will not make the comparison between first two contract 1 altenatives.

Contact 1; Contract 1; Contract 3.

Best wishes,
Yang

[Michiel Bliemer]

Ngene is optimising for efficiency to reduce standard errors in model estimation. The experimental design that is generated is driven by your choices for priors and constraints. You have imposed quite a few constraints, and also there are a lot of dominance constraints applied, which makes contract=1 appear in many more choice tasks than the other constraints. This is mainly due to your very strict constraints that plug_in_hours must be zero if contract is 2 or 3, so this attribute only gets trade-offs when contract=1. According to your priors, plug_in_hours has one of the greatest impacts on utility, so it seems a quite important attribute. It is no problem if you want to restrict the number of times that contract = 1 appears, so feel free to impose some further constraints but note that any constraints you impose reduces design efficiency.

You can also manually make changes to the design if you like, or you can generate 40 or so choice tasks and select 24 manually, and afterwards you could evaluate your design using ;eval in Ngene.

Michiel