choice-metrics.com

Hi all,

I came across a design today in which I’d like to define the levels of some attributes as -20%,-10%, 0, +10%, and some others as 15, 35, 60 (minutes). Please, note that when I say -20%, -10%, 0, +10%, I don’t mean in reference to the value of another attribute (using .ref and .piv). What I mean is -20%, -10%, 0, +10% with respect to a stated value declared in the survey previous to the SCE. For instance, a -20%, -10%, 0, or +10% of the travel time of the last trip that the respondent indicated in a previous section.

So I wonder if the utilities could be written, for instance, as:
U(Car) = b1 * tt_car[-1,0,1,2] +
b2 * tc_car[-1,0,1,2] +
b3 * ac_car[-1,0,1,2] /

U(Air) = b4 * tt_air[-2,-1,1,2] +
b5 * tc_air[-2,-1,1,2] +
b6 * ac_air[-2,-1,1,2] +
b7 * tfrt_air[15,35,60] +
b8 * HS_tfr[0,1] /


U(Bus) = b10 * tt_bus[-2,-1,0,1] +
b11 * tc_bus[-2,-1,0,1] +
b12 * ac_bus[-2,-1,0,1] +
b13 * tfrt_bus[15,35,60] /


where -2,-1,0,1 would stand for -20%, -10%, 0, +10%.

Would that be correct? My concern is about mixing values that represent a percentage of ‘modification’ with numbers that indeed represent ‘true’ values. Does what I say make sense?

Thanks for your help.

J.

This is not correct if you want to create an efficient design, you need to use the actual attribute levels that you would use in model estimation if you are generating an efficient design. With ref and piv, Ngene converts the pivots into actual attribute levels. If you are using an orthogonal design, then your specification would be OK.

In your case, since you are referring to a value in a survey instrument unknown to Ngene at the design stage, I suggest that you first create a design with actual attribute levels around AVERAGE LEVELS. For example, you could use tt_car[15,30,45] if you believe that the average travel time is 30 that respondents will state earlier in the survey. After you have created this design, you manually convert these actual levels into pivots, where you replace 15 with -50%, 30 with 0%, and 45 with 50%. This way, you can create an efficient pivot design around average attribute levels.

If you suspect a wide range of values for tt_car, you can create a library of designs instead of a pivot design. That means creating a design around 10 minutes average, a design around 20 minutes, etc. This can also be done in combination with other attributes, for example a low travel time (10 minutes) combined with a medium travel cost ($2) or something like that. This could result in a large number of designs but that is generally not an issue for survey instruments. Then whenever you read the values from the survey instrument that the respondent provided, you select the appropriate design from the library.

I typically use a library of designs because I can make sure that all designs will work properly. Implementing pivot designs in a survey instrument is usually more difficult and has some risk, for example what do you do if a respondent states a travel time of 1000, or a cost of 0 (so that you can get negative cost)? To avoid such issues, you would need to implement logic into the survey instrument. This can be avoided using a library of designs.

For more about pivot designs, you may also want to read Rose et al. (2008).

Rose, J.M., M.C.J. Bliemer, D.A. Hensher, and A. Collins (2008) Designing Efficient Stated Choice Experiments in the Presence of Reference Alternatives. Transportation Research Part B, Vol. 42, pp. 395-406.

Michiel

Michiel,

Thanks for your reply and the reference. It’s been very useful to know how you proceed because I usually create designs with actual levels, and not levels referred to a survey instrument. In any case, I hope you don’t mind if I make a couple of more questions.

I understand your suggestion of creating first an efficient design around average levels and then replacing the absolute values with the percentages manually. Yet, when you suggest doing the same for a range of values you don't mean to do so and then replace them with the percentages, do you? My point is that if you do an efficient design for tt_car 5,10,15 minutes, and another one for tt_car 10,20,30, for instance, in both cases those would be -50%,0,+50% relative levels, and both designs would be equivalent. Am I wrong?

My second question has to do with why my specification would be OK in an orthogonal design and not in an efficient one? In other words, why an efficient design needs the actual levels… or why an orthogonal design can include a mix of relative and actual values?
Related to this (orthogonal/efficient), I wanted to include some conditions in the design that I showed above. Conditions are not allowed in orthogonal designs, so I added them and indicated an efficient design without priors. That would be equivalent to an orthogonal design, allowing for the use of the ;cond clause, right?

Finally, this is a question that I always had (although I know it can be a different, long conversation): in general, which are the potential problems that creating a bad design could entail in subsequent phases (modeling)? I mean, mixing up relative and actual values in an efficient design as I was suggesting above, not using an appropriate range for the levels (or not wide enough), defining an MNL in the design when one afterward decides that an MMNL is better….

Thanks again for your time.

BR,

J.

To answer your first question, yes you are wrong. Creating an efficient design with levels 5,10,15 will result in a DIFFERENT design than when using levels 10,20,30, despite replacing the levels with percentages. An orthogonal design is different from an orthogonal design or a random design, despite the attribute levels being the same.

To answer your second question, orthogonal designs are not efficient and do not care what the attribute levels are. This is an inferior design technique and no matter what levels you use, you get the same design. An efficient design is a smarter design technique that maximises information captured in your survey and requires knowledge of attribute levels and parameter priors to optimise information.

Orthogonal designs are imcompatible with any constraints (cond, require, reject, dominance checks, etc). Constraints can only be imposed on efficient designs since they are more flexible. An efficient design with zero priors is not orthogonal, but it is more similar to an orthogonal design since parameter prior information is ignored. I would be comfortable in generating a single efficient design with zero priors and using average attribute levels, then converting to percentages and using those for any input from the respondent. When using zero priors, there is no real need to generate multiple designs. And then you can also apply your conditional constraints, so what you suggest is an acceptible way forward I think.

To answer your last question, a bad design has one or more of the following characteristics:

1. (near) perfect correlation across attributes
2. too little variation in the attribute levels (too narrow range)
3. strictly dominant alternatives where one alternative has better levels for all attributes, which captures zero information
4. weakly dominant alternatives that have a very high choice probabiliy (e.g. 95-99%), which does not capture much information; not that this assessment requires parameter priors
5. unrealistic attribute level combinations, which is why people often impose constraints
6. having too few rows in the design; one requires sufficient variation in the data to estimate more complex models like MMNL afterwards

Given that it is near-impossible to optimise a design for MMNL or other complex model types, it is common to design the data for the MNL model. Optimising for the MNL model is already MUCH better than using an orthogonal or random design, and such a design will generally also work well for more complex models (MMNL is still quite similar to MNL in many ways, especially in terms of choice probabilities).

Michiel

Thanks again for your reply. However, I still have the feeling that I’m missing a piece of information. Why do you say, When using zero priors, there is no real need to generate multiple designs? You advocated in your previous reply for generating multiple efficient designs for different levels of the instrument.

Regarding the consequences of a bad design. Would the definition of generic (specific) parameters in the design have a negative impact on estimation if, finally, one decides to estimate the parameters specific (generic) in the final model?

And just one more question about orthogonal designs, sorry if it doesn’t make much sense, I haven’t worked much with them. If an attribute has levels, let’s say, [10,15,20] and another one has [12,18,24]; then they should be coded as b1*A[0,1,2] + b2*B[0,1,2] (because they have 3 levels each), or as b1*A[0,1,2] + b2*B[3,4,5] (because they have three levels each, but different among them)? Moreover, since A[0,1,2] would be just labels, would that be the same as defining A[–1,0,1]? Sorry, I’m a bit confused with this. Since the inner product of two columns in an orthogonal design is zero, it is my understanding that the actual numbers that you write down do matter.

BR,

J.

People usually generate designs with non-zero priors, so in that case you would generate multiple designs. But with zero priors the attribute levels do not matter much anymore since they are all multiplied with 0 in the utility function. It is up to you if you want to use zero or non-zero priors. Clearly informative non-zero priors will result in a much more efficient design, but setting them to zero is a pragmatic solution if you do not have parameter priors. I was initially not aware that you were talking about setting all priors to zero (I usually only do that for a pilot study but not for the main study).

If I am unsure about estimating generic or alternative-specific parameters then I usually assume that they are alternative-specific when I create the design, which will make sure that I can estimate the model both with alternative-specific and generic parameters. If you optimise your design for alternative-specific parameters but you end up estimating generic parameters, then you lose some efficiency. Similarly the other way around. In rare cases you may not be able to estimate alternative-specific parameters if you assumed generic parameters in the design phase, especially if you use a design with a small number of rows.

You are correct that levels for orthogonal designs are just placeholders, also referred to as symbols, so you can use any of the levels that you wrote, 0,1,2 or 3,4,5 or -1,0,1. The orthogonal design will be exactly the same. Orthogonality in experimental design is defined as each pair of attribute levels appearing an equal number of times across the choice tasks. This definition does not require the inner product to be zero, it is a more general definition.

Michiel

Great, thanks for all your answers.