Experiments and context variables: efficient design

[peyman_07]

Dear Ngene team,

First of all, I would like to thank you for providing a comprehensive manual and this valuable platform to support NGENE users.

I am fairly new to Ngene. I have read the manual and the forum topics, but I have still some questions. I am investigating the electric vehicle users' route choice and charging behavior. There are six route alternatives in my study: freeway with/without charger, arterial way with/without charger, local streets with/without charger. Battery level at the origin (SOC) and availability of charging point at the destination are two important factors to select a route. There are three possibilities based on the level of battery at the origin and electricity consumption: Green zone (No need to charge during the trip), Grey Zone (charge or not?), Red Zone (need to charge). So, I have defined three different experiments with different battery level at the origin and availability of charger at the destination as context variables. Following please find these three designs:

(SOC: State-of-Charge (battery level at the origin)
CP_D: Charging point at the destination
CONS: percentage of electricity consumption for this trip
TT: Travel time
TC: Travel cost
CL: Charger location
CT: Charging time
WT: Waiting time in line for charging)

E1:
design
;alts=F1,A1,L1,F2,A2,L2
;rows=24
;block=6
;eff=(mnl,d)
;model:

U(F1)=soc[0.000001]*SOC[70,80,90]+cp_d[0.000001]*CP_D[0,1]+cons[-0.000001]*CONS[20,30,40]+tt1[-0.000001]*TT[10,15,20]+tc[-0.000001]*TC[2,4,6]/
U(A1)=soc*SOC[SOC]+cp_d*CP_D+cons*CONS+tt2*TT+tc*TC/
U(L1)=soc*SOC[SOC]+cp_d*CP_D+cons*CONS+tt3*TT+tc*TC/
U(F2)=cons*CONS+tt4*TT+tc2*TC2[6,8,10]+CL.dummy[-0.000001|-0.000001]*CL[0,1,2]+ct[-0.000001]*CT[20,25,30]+wt[-0.000001]*WT[0,5,10]/
U(A2)=cons*CONS+tt5*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT/
U(L2)=cons*CONS+tt6*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT

$

E2:
design
;alts=F1,A1,L1,F2,A2,L2
;rows=24
;block=6
;eff=(mnl,d)

;cond:
if(f1.soc=40,f1.cons=35 or f1.cons=40),
if(a1.soc=40,a1.cons=35 or a1.cons=40),
if(l1.soc=40,l1.cons=35 or l1.cons=40)



;model:

U(F1)=soc[0.000001]*SOC[40,45,50]+cp_d.dummy[0.000001]*CP_D[0,1]+cons[-0.000001]*CONS[35,40,45]+tt1[-0.000001]*TT[15,20,25]+tc[-0.000001]*TC[4,6,8]/
U(A1)=soc*SOC[SOC]+cp_d.dummy*CP_D[CP_D]+cons*CONS+tt2*TT+tc*TC/
U(L1)=soc*SOC[SOC]+cp_d.dummy*CP_D[CP_D]+cons*CONS+tt3*TT+tc*TC/
U(F2)=cons*CONS+tt4*TT+tc2*TC2[8,9,10]+CL.dummy[-0.000001|-0.000001]*CL[0,1,2]+ct[-0.000001]*CT[20,25,30]+wt[-0.000001]*WT[0,5,10]/
U(A2)=cons*CONS+tt5*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT/
U(L2)=cons*CONS+tt6*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT

$

E3:
design
;alts=F2,A2,L2
;rows=24
;block=6
;eff=(mnl,d)


;model:

U(F2)=soc[0.000001]*SOC[20,30,40]+cp_d.dummy[0]*CP_D[0,1]+cons*CONS[40,45,50]+tt1*TT[20,25,30]+tc*TC2[8,9,10]+CL.dummy[-0.000001|-0.000001]*CL[0,1,2]+ct[-0.000001]*CT[20,25,30]+wt[-0.000001]*WT[5,10,15]/
U(A2)=soc*SOC[SOC]+cp_d.dummy*CP_D[CP_D]+cons*CONS+tt2*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT/
U(L2)=cons*CONS+tt3*TT+tc2*TC2+CL.dummy*CL+ct*CT+wt*WT

$




Here are my questions:

As mentioned, I have defined three experiments that each of them has two context variables. Based on the number of rows and blocks, each respondent faces 4 choice tasks from each experiment and in total 12 choice sets for each respondent.

1- Can you please confirm the feasibility of this design to ensure I am in the right direction?

2- Is it possible to estimate all experiments in one model?

3- I treated SOC and CP_D as context variables. I put them in three routes without charger to explore the impact of battery level and availability of charger at the destination on selecting routes with/without charger. Is it correct? If yes, how can I interpret this parameter? for example, let's assume B_soc=+0.0123. How can I interpret this parameter in the final model?

4- As far as I know, the minimum number of rows is #parameters/#alternatives-1. For the first experiment, for instance, there are 13 parameters and 6 alternatives. So the minimum number of rows is (13/(6-1))+1~4. Are 4 rows enough? What would happen if the number of rows increases? How many rows are required to end up with the most efficient design? Is there any formula?

5- I designed three different experiments. So, Ngene says which choice sets must be presented to each respondent per each experiment (separately). How can I combine the selected choice sets from each experiment to present to each respondent? Randomly?
Is there any possibility in Ngene to put all of them in the software and Ngene specifies just one design that states which choice sets in all experiments should be presented to each respondent?


6- I can estimate three different models. Is it possible to compare the parameters in these experiments with each other? For example, is it possible to compare these two variable in two experiments: B_soc=0.002 in first experiment where SOC levels are 70,80,90% versus B-soc=0.042 in the second experiment where SOC levels are 40,45,50.
Is it possible to integrate all of them in just one model and estimate it? If yes, which one is more reliable? integrated model or separate models?

4- I have considered very small priors because I have no idea about the precise value. So, I need to conduct a pilot study. However, the S error is around 2 billion! Is it normal? How many respondents I need to obtain the priors? If any of the parameters were not significant in the pilot-study model, would I remove it from the final design or I need to incorporate its value in the design as prior?


I would appreciate your great favor to allocate your time to read such a long text and answer questions!

Looking forward to hearing from you.
Kind regards,
Peyman.

[Michiel Bliemer]

Dear Peyman,

1. Running your syntax gives a finite D-error, which means that your model can be estimated using the design generated by Ngene. Therefore, it is feasible.

2. Yes you can pool all the data and estimate a single joint model. You need to consult your estimation software on how to input different choice set sizes.

3. I can see that SOC is a scenario variable that is constant across multiple alternatives, but I am not sure why you see CP_D as a context variable since it seems to vary across alternatives and therefore looks like a regular attribute? Scenario variables can either be put into J-1 alternatives (where J is the choice set size) or can be interacted with other attributes, e.g. SOC*CONS. If soc is a positive parameter, then that means that the higher the battery level, the more likely one is to select a route without charger (alternatives F1/A1/L1).

4. Yes 4 rows is theoretically enough but this is the absolute minimum and you generally would like to have more variation in your data. If you are estimating a mixed logit model with random coefficients (normally distributed), then you need double the number of rows. If you estimate a latent class model, you may need even more. The more rows, the more variation in your data, but this is not necessarily more efficient PER CHOICE TASK. If you use more rows, you also have to increase the number of blocks, which increases sample size at the same time. There is no formula for the optimal combination of rows and blocks. All you want is sufficient variation and I think that 24 rows should give you sufficient variation in your data.

5. For efficiency it does not matter which blocks you combine and give to a respondent. But in general you would like to have some attribute level balance within each respondent, i.e. that each respondent sees some high and low values of each attribute.

6. Maybe it is possible to put this all into a single Ngene syntax. The ;fisher command may be useful here (although this may not work with constraints).

Michiel

[peyman_07]

Dear Peyman,

1. Running your syntax gives a finite D-error, which means that your model can be estimated using the design generated by Ngene. Therefore, it is feasible.

2. Yes you can pool all the data and estimate a single joint model. You need to consult your estimation software on how to input different choice set sizes.

3. I can see that SOC is a scenario variable that is constant across multiple alternatives, but I am not sure why you see CP_D as a context variable since it seems to vary across alternatives and therefore looks like a regular attribute? Scenario variables can either be put into J-1 alternatives (where J is the choice set size) or can be interacted with other attributes, e.g. SOC*CONS. If soc is a positive parameter, then that means that the higher the battery level, the more likely one is to select a route without charger (alternatives F1/A1/L1).

4. Yes 4 rows is theoretically enough but this is the absolute minimum and you generally would like to have more variation in your data. If you are estimating a mixed logit model with random coefficients (normally distributed), then you need double the number of rows. If you estimate a latent class model, you may need even more. The more rows, the more variation in your data, but this is not necessarily more efficient PER CHOICE TASK. If you use more rows, you also have to increase the number of blocks, which increases sample size at the same time. There is no formula for the optimal combination of rows and blocks. All you want is sufficient variation and I think that 24 rows should give you sufficient variation in your data.

5. For efficiency it does not matter which blocks you combine and give to a respondent. But in general you would like to have some attribute level balance within each respondent, i.e. that each respondent sees some high and low values of each attribute.

6. Maybe it is possible to put this all into a single Ngene syntax. The ;fisher command may be useful here (although this may not work with constraints).

Michiel

Dear Prof. Michiel Bliemer

Many thanks for your response.

As mentioned, CP_D is a context variable but I had forgotten to fix it across alternatives. I redefine it thanks to your advice. Apart from that, I put many constraints in the syntax as follows:

design
;alts=F1,A1,L1,F2,A2,L2
;rows=24
;block=6
;eff=(mnl,d)


;cond:
if(f1.cons1<=a1.cons2,f1.cons1>a1.cons2),
if(f1.cons1<=a2.cons2,f1.cons1>a2.cons2),
if(f1.cons1<=l1.cons3,f1.cons1>l1.cons3),
if(f1.cons1<=l2.cons3,f1.cons1>l2.cons3),
if(f2.cons1<=a1.cons2,f2.cons1>a1.cons2),
if(f2.cons1<=a2.cons2,f2.cons1>a2.cons2),
if(f2.cons1<=l1.cons3,f2.cons1>l1.cons3),
if(f2.cons1<=l2.cons3,f2.cons1>l2.cons3),
if(a1.cons2<=l1.cons3,a1.cons2>l1.cons3),
if(a1.cons2<=l2.cons3,a1.cons2>l2.cons3),
if(a2.cons2<=l1.cons3,a2.cons2>l1.cons3),
if(a2.cons2<=l2.cons3,a2.cons2>l2.cons3),

if(f1.tt1>=a1.tt2,f1.tt1<a1.tt2),
if(f1.tt1>=a2.tt2,f1.tt1<a2.tt2),
if(f1.tt1>=l1.tt3,f1.tt1<l1.tt3),
if(f1.tt1>=l2.tt3,f1.tt1<l2.tt3),
if(f2.tt1>=a1.tt2,f2.tt1<a1.tt2),
if(f2.tt1>=a2.tt2,f2.tt1<a2.tt2),
if(f2.tt1>=l1.tt3,f2.tt1<l1.tt3),
if(f2.tt1>=l2.tt3,f2.tt1<l2.tt3),
if(a1.tt2>=l1.tt3,a1.tt2<l1.tt3),
if(a1.tt2>=l2.tt3,a1.tt2<l2.tt3),
if(a2.tt2>=l1.tt3,a2.tt2<l1.tt3),
if(a2.tt2>=l2.tt3,a2.tt2<l2.tt3),

if(f1.tc1<=a1.tc2,f1.tc1>a1.tc2),
if(f1.tc1<=l1.tc3,f1.tc1>l1.tc3),
if(f1.tc1>=f2.tc4,f1.tc1<f2.tc4),



if(f2.tc4<=a2.tc5,f2.tc4>a2.tc5),
if(f2.tc4<=l2.tc6,f2.tc4>l2.tc6),

if(a1.tc2<=l1.tc3,a1.tc2>l1.tc3),
if(a1.tc2>=a2.tc5,a1.tc2<a2.tc5),

if(a2.tc5<=l2.tc6,a2.tc5>l2.tc6),

if(l1.tc3>l2.tc6,l1.tc3<l2.tc6),

if(f2.ct1<=a2.ct2,f2.ct1>a2.ct2),
if(f2.ct1<=l2.ct3,f2.ct1>l2.ct3),
if(a2.ct2<=l2.ct3,a2.ct2>l2.ct3)


;model:

U(F1)=soc[0.000001]*SOC[70,80,90]+cp_d[0.000001]*CP_D[0,1]+cons[-0.000001]*CONS1[25,30,35]+tt1[-0.000001]*TT1[10,15,20]+tc*TC1[3,4,5]/
U(A1)=soc*SOC[SOC]+cp_d*CP_D[CP_D]+cons*CONS2[20,25,30]+tt2[-0.000001]*TT2[15,20,25]+tc*TC2[2,3,4]/
U(L1)=soc*SOC[SOC]+cp_d*CP_D[CP_D]+cons*CONS3[15,20,25]+tt3[-0.000001]*TT3[20,25,30]+tc*TC3[1,2,3]/
U(F2)=cons*CONS1+tt4[-0.000001]*TT1+tc2[-0.000001]*TC4[4,5,6]+CL.dummy[-0.000001|-0.000001]*CL[0,1,2]+ct[-0.000001]*CT1[15,20,25]+wt[-0.000001]*WT[0,5,10]/
U(A2)=cons*CONS2+tt5[-0.000001]*TT2+tc2*TC5[3,4,5]+CL.dummy*CL+ct*CT2[10,15,20]+wt*WT/
U(L2)=cons*CONS3+tt6[-0.000001]*TT3+tc2*TC6[2,3,4]+CL.dummy*CL+ct*CT3[5,10,15]+wt*WT

$


This is just one out of three experiments. As mentioned, I would like to present 4 choice sets in each experiment to respondents (in total 12 choice tasks per respondent). I would be grateful if you answer the following questions:

1- Although Ngene can run this model, do these constraints lead to losing efficiency of design?

2- Do these constraints have any effect on the final model (e.g., making some parameters insignificant)?

3- I need to obtain priors so I am going to conduct a pilot study. How many respondents are required for this design? If any parameter was insignificant in the MNL model, would I remove the associated attribute from the final design? Is significance level important in the pilot study?

Many thanks for your great help!

Looking forward to hearing from you.
Kind regards,
Peyman.

[Michiel Bliemer]

1. Putting constraints on the design will always lead to a loss of efficiency. But realism of a choice task is more important than efficiency, so if the constraints are needed for the choice tasks to make sense then you should include them. If they are not needed to make sense, then you should preferably avoid using them.

2. No, as long as your model is still identifiable. I ran your syntax and it generates a design with a finite D-error, which means that you can estimate all parameters. Only if a D-error becomes infinite ("undefined" in Ngene) then there is a problem.

3. I cannot tell you how many respondents you need, you have to make guesstimate yourself. You are estimating 15 parameters, think about how many data points you would need to estimate these parameters to a low level of statistical significant (eg instead of 95% level maybe 80% level). Maybe a few hundred? Let's say 300. This means 300/24 = 12.5 design replications, which with 6 blocks means 75 respondents. If your budget or sample allows less, then select less. This is all guessing, there is no scientific unpinning here. Your parameters need not be statistically significant in a pilot study, you can use Bayesian priors with a wide distribution to account for the unreliability of the priors. You do NOT remove attributes with insignificant parameters (from your pilot study) from your final design since they may become significant with larger sample sizes.

Michiel

[peyman_07]

Dear Prof. Michiel Bliemer

Many thanks for your quick and complete response.

If one or several alternatives are infeasible in some scenarios, what shall we need to do?

For example, when the battery level at the origin is 50% and the consumption is 60%. In this case, a person cannot select a route without fast charging because he/she cannot arrive at his/her destination so routes without charging are infeasible. Please assume that we are not going to impose any constraint between the battery level at the origin and consumption. If an alternative is infeasible in some scenarios (choice sets), does the design still work and give valid results?

My second question is about the importance of attribute level balance in each block. In this survey, I have considered 24 choice sets in 6 blocks. Some attributes have 3 levels and some of them have 2 levels. It means that we do not have attribute level balance in each block for the ones having three levels (4 choice tasks in each block). Does it have any influence on the result?

Your advice is always appreciated!

Kind regards,
Peyman.

[Michiel Bliemer]

1. If you are not imposing constraints and some attribute level combinations are not feasible, then statistically your design will still work and you can still estimate a model, but I would doubt that you can appropriate choice responses. If alternatives do not make sense to respondents, I think one cannot assume that they make logical choices. They may simply think that the survey makes no sense and respond randomly. Therefore, it is important to avoid attribute level combinations that do not make sense by imposing constraints in the design (or to manually change attribute levels if needed).

2. Only with orthogonal designs it is possible to guarantee attribute level balance within a block. In efficient designs it is not possible to guarantee this. Attribute level balance is often assumed a nice property, but it is not a mandatory property. Statistically there is no influence on your model estimations. Behaviourally, one may argue that a person only seeing low levels may respond differently to a person that sees a mix of low and high levels. Therefore, some degree of attribute level balance is usually nice to have.

Michiel

[peyman_07]

Dear Prof. Michiel Bliemer

Thanks for your email.

I added three more constraints into the syntax, but I received the following alert:

"The conditional statement specified may require the generation within Ngene of a very large factorial design for the attributes associated with the statement. a sudden crash may result due to a shartage of memory. If this happens, add in the following to your comma separated list of conditional statements: "fractional=X%". This will generate a fractional factorial that is roughly X% of the full factorial size. A low enough percentage should solve the memory problem. One of the factorial designs has 531441 rows."

Please find the full syntax below:
design
;alts=F1,A1,L1,F2,A2,L2
;rows=24
;block=6
;eff=(mnl,d)


;cond:
if(f1.cons1<=a1.cons2,f1.cons1>a1.cons2),
if(f1.cons1<=a2.cons2,f1.cons1>a2.cons2),
if(f1.cons1<=l1.cons3,f1.cons1>l1.cons3),
if(f1.cons1<=l2.cons3,f1.cons1>l2.cons3),
if(f2.cons1<=a1.cons2,f2.cons1>a1.cons2),
if(f2.cons1<=a2.cons2,f2.cons1>a2.cons2),
if(f2.cons1<=l1.cons3,f2.cons1>l1.cons3),
if(f2.cons1<=l2.cons3,f2.cons1>l2.cons3),
if(a1.cons2<=l1.cons3,a1.cons2>l1.cons3),
if(a1.cons2<=l2.cons3,a1.cons2>l2.cons3),
if(a2.cons2<=l1.cons3,a2.cons2>l1.cons3),
if(a2.cons2<=l2.cons3,a2.cons2>l2.cons3),

if(f1.tt1>=a1.tt2,f1.tt1<a1.tt2),
if(f1.tt1>=a2.tt2,f1.tt1<a2.tt2),
if(f1.tt1>=l1.tt3,f1.tt1<l1.tt3),
if(f1.tt1>=l2.tt3,f1.tt1<l2.tt3),
if(f2.tt1>=a1.tt2,f2.tt1<a1.tt2),
if(f2.tt1>=a2.tt2,f2.tt1<a2.tt2),
if(f2.tt1>=l1.tt3,f2.tt1<l1.tt3),
if(f2.tt1>=l2.tt3,f2.tt1<l2.tt3),
if(a1.tt2>=l1.tt3,a1.tt2<l1.tt3),
if(a1.tt2>=l2.tt3,a1.tt2<l2.tt3),
if(a2.tt2>=l1.tt3,a2.tt2<l1.tt3),
if(a2.tt2>=l2.tt3,a2.tt2<l2.tt3),

if(f1.tc1<=a1.tc2,f1.tc1>a1.tc2),
if(f1.tc1<=l1.tc3,f1.tc1>l1.tc3),
if(f1.tc1>=f2.tc4,f1.tc1<f2.tc4),



if(f2.tc4<=a2.tc5,f2.tc4>a2.tc5),
if(f2.tc4<=l2.tc6,f2.tc4>l2.tc6),

if(a1.tc2<=l1.tc3,a1.tc2>l1.tc3),
if(a1.tc2>=a2.tc5,a1.tc2<a2.tc5),

if(a2.tc5<=l2.tc6,a2.tc5>l2.tc6),

if(l1.tc3>l2.tc6,l1.tc3<l2.tc6),

if(f2.ct1<=a2.ct2,f2.ct1>a2.ct2),
if(f2.ct1<=l2.ct3,f2.ct1>l2.ct3),
if(a2.ct2<=l2.ct3,a2.ct2>l2.ct3),

if(f2.tt1+f2.ct1+f2.wt>a2.tt2+a2.ct2+a2.wt,f2.tt1+f2.ct1+f2.wt<=a2.tt2+a2.ct2+a2.wt),
if(f2.tt1+f2.ct1+f2.wt>l2.tt3+l2.ct3+l2.wt,f2.tt1+f2.ct1+f2.wt<=l2.tt3+l2.ct3+l2.wt),
if(a2.tt2+a2.ct2+a2.wt>l2.tt3+l2.ct3+l2.wt,a2.tt2+a2.ct2+a2.wt<=l2.tt3+l2.ct3+l2.wt)











;model:

U(F1)=soc[0.000001]*SOC[70,80,90]+cp_d[0.000001]*CP_D[0,1]+cons[-0.000001]*CONS1[20,25,30]+tt1[-0.000001]*TT1[20,25,30]+tc*TC1[3,3.5,4]/
U(A1)=soc*SOC[SOC]+cp_d*CP_D[CP_D]+cons*CONS2[15,20,25]+tt2[-0.000001]*TT2[25,30,35]+tc*TC2[2,2.5,3]/
U(L1)=soc*SOC[SOC]+cp_d*CP_D[CP_D]+cons*CONS3[10,15,20]+tt3[-0.000001]*TT3[30,35,40]+tc*TC3[1,1.5,2]/
U(F2)=cons*CONS1+tt4[-0.000001]*TT1+tc2[-0.000001]*TC4[5,5.5,6]+CL.dummy[-0.000001|-0.000001]*CL[0,1,2]+ct[-0.000001]*CT1[8,10,12]+wt[-0.000001]*WT[0,3,6]/
U(A2)=cons*CONS2+tt5[-0.000001]*TT2+tc2*TC5[4,4.5,5]+CL.dummy*CL+ct*CT2[6,8,10]+wt*WT/
U(L2)=cons*CONS3+tt6[-0.000001]*TT3+tc2*TC6[3,3.5,4]+CL.dummy*CL+ct*CT3[4,6,8]+wt*WT

$

Can you please let me know what this notification means? Does it mean that the number of constraints is too much?

Thanks for your support.
Kind regards,
Peyman.

[Michiel Bliemer]

Sorry for the late reply, I seem to have missed this post.

Since you are asking Ngene to use the default swapping algorithm using ;cond constraints, instead of asking Ngene to use the modified Federov algorithm using ;require or ;reject constraints, Ngene aims to maintain attribute level balance and as such it creates the full factorial structure of the constraint set. The more constraints, the more combinations Ngene needs to consider in order to keep attribute level balance. Using a lot of constraints is always very tricky since it makes it more difficult to find feasible designs (and in some cases, none exist). Usually the modified Federov algorithm can handle more constraints because it ignores attribute level balance, while conditional constraints in the swapping algorithm are more limited and adding a lot of them can create issues. We typically recommend only adding constraints that are absolutely necessary, i.e. constraints that are needed to create attribute combinations that make sense.

If the number of constraints turns out to be problematic, then of course it is always possible to use a smaller number of them and make manual adjustments to your design afterwards. You can evaluate a manually adjusted design in Ngene via the ;eval command.

Michiel