Alternative specific efficient design

[Pauline]

Hi Choice Metrics Team,

I am trying to generate a design for a freight mode choice survey and therefore need to use a labeled alternative specific design.
I included priors based on studies but I’m planning to do a pilot study to confirm the assumptions. Here is the current design:

design
;alts = A,B,C
;rows = 32
;block = 4
;eff = (mnl,d)
;model:
U(A) = b1[-0.0051] * RateA[4400,4900,5600]+ b2[0.0245]* FrequA[7,10,15] + b3[-0.0325] * TimeA[19,24,36] + b4[0.0457] * ReliaA[1,5] /
U(B) = b1 * RateB[1100,1300,1500] + b2 * FrequB[2,5,6]+ b3 * TimeB[30,36,48]+ b4 * ReliaB[0,3] /
U(C) = b1 * RateC[2700,3000,3300]+ b2 * FrequC[2,5,8]+ b3 * TimeC[48,60,72] +b4 * ReliaC[2,4]
$

MNL efficiency measures are: D-error: 0.110073, A-error: 2.683779, B-estimate: 2E-06, S-estimate: 15361.698928. There is obviously something wrong with this design. I tested it with slightly different attribute levels and priors but could not generate a major improvement on the efficiency of the design.

I would be very grateful for any comments on the design.

Cheers,
Pauline

[Michiel Bliemer]

Hi Pauline,

Yes there is indeed something wrong. You can see that the B estimate (balancedness) is almost zero, which indicates that a single alternative is always chosen. You can confirm this by looking at the choice probabilities for each choice task under Design properties, MNL, and then Probabilities.

The reason is that your prior for Rate is too large (in an absolute sense), making Rate the dominant attribute and hence alternative B the dominant alternative. You can see this by looking at the contribution of each attribute to utility. For Rate there are the following contributions (using the middle attribute level):
A: -0.0051 * 4900 = -24.99
B: -0.0051 * 1300 = -6.63
C: -0.0051 * 3000 = -15.3
All other attributes have much smaller contributions to utility (and their priors look fine). The difference between A, B, and C is very large (utility differences are normally somewhere between 0 and 2, where 2 is already quite large). This indicates that b1 = -0.0051 is likely incorrect and inconsistent with your rates.

if you think that the values for Rates are fine (which I assume), then something is wrong with the prior value of -0.0051, Where did this value come from? Were the rates in the other study in a different currency? Or another unit (dollar cents instead of dollars)? Or did you forget to put in an extra 0 (b1 = 0.00051 works fine)? Or did the other study use Rates that are much lower (i.e. 49 instead of 4900)?

Michiel

[Pauline]

Hi Michel,

Thank you very much for your helpful comments!

The Rate levels are realistic value ranges for the corridor that I am investigating. I got these values confirmed by the respective industries/modes, so the attribute levels in my design should be ok.

For the priors I used parameters from other studies that investigated very similar issues and divided them by 2. These studies used the same unit for Rate (Dollars). I’m relatively sure about the sign of the parameters based on general perception of parameters and the existing literature. The scale of the parameters seems to be an issue though. I noticed that it is recommended to used parameter ratios from the literature rather than simply the parameters from other studies. What exactly is meant by parameter ratios?

Given that I got a relatively efficient design here (with the correction of the one digit for the Rate prior), would you suggest to use that design or still run a pilot to get more clarity about the more realistic priors for the circumstances of my study?

How would you go about eliciting prior information from a pilot study? Would you use any design, say zero priors, get your responses, estimate the parameters and use them as priors for the design?

Thank you in advance for your advice.

Cheers,
Pauline

[Michiel Bliemer]

Is may be a scaling issue yes, the coefficient is very large relative to the rates, it seems like a mistake as I do not believe it would lead to a sensible model unless their range was much more narrow (and not ranging from 1000 to 5000).

Transferring priors has to be done carefully because we do not know the scale. In logit models you estimate lambda*beta1, lambda*beta2, etc., where it is not possible to distinguish lambda from the betas, therefore typically lambda is normalised to 1. If you take the ratios, (lambda*beta1) / (lambda*beta2) = beta1/beta2, then it is clear that the ratios are not sensitive to lambda. So while we cannot simply transfer the betas, we can transfer the beta ratios. Since we do not know lambda, it is best to use a conservative scale (like lambda = 0.5 as you have done, or even closer to zero).

My approach to obtaining priors is typically the following:
1. Determine the sign of each coefficient and use a small value for each prior indicating the design (e.g., -0.000001 or +0.000001), which can be used to detect dominant alternatives in Ngene (using ;alts = A*,B*,C*)
2. Estimate betas on the pilot study data and also obtain the standard errors (se)
3. Use normally distributed Bayesian priors N(lambda*beta,lambda*se) to create a Bayesian efficient design, where lambda<1 (e.g., lambda = 0.5)

Although you mention that you are using a labelled experiment, since you have generic coefficients and each alternative has the same attributes, and also since you are not using any constants, this is actually an unlabelled experiment with generic alternatives, instead of alternative specific alternatives. That your attribute levels are different across alternatives does not make it alternative specific. So I would use the method outlined in step 1 to check for dominant alternatives.

You can also use your own (expert) judgement to obtain Bayesian priors. For more information about determining priors, please refer to the short technical note we recently published in the Journal of Choice Modelling:

Bliemer & Collins (2016) On determining priors for the generation of efficient stated choice experimental designs. Journal of Choice Modelling, in press.

Michiel

[Pauline]

Thank you, Michiel!

Your comments are very helpful.

Cheers,
Peggy