Vary scenarios among respondents

[Baiba]

Dear Ngene moderators,

Just as many others, I have been finding this forum so helpful. Thank you for putting the effort in explaining all questions in depth!

I have a simple situation: I would like to have a design where scenarios are assigned to respondents, but do not vary for a given respondent. I am aware of the 'cost' conceptually, that the responses to these scenarios will be correlated to the respondents' differences, but I believe that our sample size will even those out. The benefit, in this case, is avoiding the confusion about the context...

So, how do I do that? As far as I see, Section 8.5 of the manual covers the case of scenarios changing within each survey. I did not find the instruction of my case elsewhere in the manual or here (please, let me know, if I missed it!).

So far my best idea: create a design without scenario attributes, block the design, and assign each block a different scenario.
However, the scenario enters the utility both as a main and as an interaction effect. I wonder if I may run into trouble estimating the interactions if I do not optimise the design with it?
I am creating an efficient design otherwise.

How do you recommend to do this (conceptually and technically)?

Thanks already,
Baiba

[Michiel Bliemer]

I think that you could follow two strategies:

1. Create a design without scenario attributes, and for each respondent (with a specific scenario) randomly assign choice tasks from this design.
2. Create a design with scenario attributes, after which you split the design into groups of specific scenarios and assign each respondent a group of choice tasks with a specific scenario.

In both cases, I believe that you will not have any issues with estimating interaction effects because in strategy 1 you have a lot of variation in your data because of the random assignment of choice tasks to scenarios, and in strategy 2 you specifically generated the experimental design to estimate the relevant main effects and interaction effects with the scenario variables.

To illustrate strategy 2, if I include scenario variable 'weather' with 3 levels (sunny, rainy, windy) and ensure that this variable has balanced levels across the design, then I can split the choice tasks into three groups, namely choice tasks with sunny weather, choice tasks with rainy weather, and choice tasks with windy weather. You can then assign choice tasks to respondents according to these groups, either randomly or by (manually) blocking the groups of choice tasks. If you have multiple scenario variables, you may want to generate a reasonably large experimental design.

Michiel

[Baiba]

Thank you, Michiel. This answers completely.

[Baiba]

Dear Michiel,

Following up, another layer of scenarios in our experiment is a policy framing, where half of the respondents will be asked to 'opt in' (and pick someone up along the way), and the other half to 'opt out' of the same option.
We want to have equivalent designs, so that we can compare market shares per question (and observe loss aversion both descriptively and through model estimates).
Practically, this results in the same design presented with the opposite signs in these scenarios.
Modelling wise, more interaction effects are interesting for the 'opt out' scenario; otherwise the utility functions are the same.

Which would you consider to be the most appropriate way to implement this?
My current ideas:
1) With model averaging property, see below. However, we would estimate the two models not on the entire sample but on the corresponding assigned segments.
2) With homogeneous pivot or covariate designs. However, it seems that this implementation is not supported with different utility functions (Error: 'The model 'shortest' that belongs to the 'fish' ;fisher specification is inconsistent with the first model. Attributes have been defined in an inconsistent order, or have inconsistent names.')
3) Designing the experiment for the more complex 'opt out' scenario, because the 'opt in' scenario is essentially a constrained version of it (with expected opposite parameter signs).

Would you suggest yet another approach to this task?

I have currently implemented 1) the model averaging option as follows:

Code: Select all

Design
;alts(Pick_up) = status_quo, presented
;alts(Shortest) = status_quo, presented
;rows = 60
;eff = Pick_up(mnl,d)+Shortest(mnl,d)
;block = 2;


? [...]

?Alt2 = presented option to pick someone up with increasing travel time, costs, etc.
;model(Pick_up):
U(status_quo) = 0 /
U(presented) = b_TT_Pick_up[-0.00001]*TT[0,20,40,60] + b_TC_Pick_up[-0.00001]*TC[0,20,40,60]
+ [...]

?Alt2 = presented option to skip the pick up and choose a shorter route with less tt, costs
;model(Shortest):
U(status_quo) = 0 /
U(presented) = b_TT_Shortest[0.00001]*TT[0,20,40,60] + b_TC_Shortest[0.00001]*TC[0,20,40,60]
+ [...]
$



[Michiel Bliemer]

Model averaging or using a homogeneous design may be worth trying. For a homogeneous design to work, you need to specify the same parameter names but you can use different priors, so btt[-0.0001]*TT in Pickup model and btt[0.0001]*TT in Shortest model. I note that you can simply use zero priors because there is no issue with dominance in a labelled experiment and hence the signs are not important. So in that case, the priors would be zeros in both models. I am not sure if you can omit attributes in one of the models when using a homogeneous design, I believe you can but I am not sure. If so, the first model needs to be the most complete model that includes all attributes, while the second model can include a subset of attributes.

Michiel

[Baiba]

Dear Michiel,

Thank you very much again!
Could you please elaborate further your advice about zero priors? I understand that dominance is not an issue for labelled experiments. And I am thinking, is there no other (efficiency) benefit provided by non-zero priors?

If there is no other benefit, then creating a single design (my option 3 before) seems like the way to go.

Baiba

[Michiel Bliemer]

A value of -0.0001 is essentially the same as a value of +0.0001, both are essentially 0. Only if you have informative priors then it makes sense to indicate the sign. So in your case you may just want to generate a design with zero priors, which will resemble an orthogonal design (which is another option you can consider).

Michiel

[Baiba]

Dear Michiel,

Thank you, I get it now. We will use a zero-prior/orthogonal design for the pilot study.
Your advise on the more involved options will be very useful for the main study.

Thank you very much for your help!

Baiba