fixed levels within choice sets for some attributes

[LauraWall]

Dear NGene team

I am working on a design with 8 attributes and two alternatives, see below (number of rows and blocks TBD based on efficiency and S estimates from design to match practical recruitment constraints):

Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/

U(alt2) = bword*word + bproc*proc +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben + bcross*cross $

I want the procedure (proc) and benefit (ben) attributes to have a fixed level within choice sets but this level vary across choice sets
I have come across a number of possible ways to achieve this but each have issues that cause me concern and I would like some advice over the specific issues within each and also advice over which approach is most appropriate
Option1: create the design in Ngene as if the levels do change for each alternative (syntax as above) and then evaluate the design but with the require property by adding the following lines before ;model:

;eval= Design.csv / Design.ngd
;require:
alt1.proc = alt2.proc,
alt1.ben = alt2.ben

This appears to work (using either the Ngene saved design, or the output from the design saved into a csv and then uploaded) in the sense that I do not get any errors – however the D error, S estimate and all other information (including the design itself) is exactly the same whether I have or do not have the require argument.
I therefore assume that it is not doing what it should, as surely requiring this should change something about the design right?

Option 2: use an availability / partial profiles design by developing the possible choice sets in another program such as R (with the levels for these attributes fixed for each choice set), and then uploading these possible choice sets and selecting an appropriate design for them by adding the following line before ;model:

;alg = mfederov(candidates = explicitPartialProfiles.csv)

Again, this works in the sense that it does not produce an error but it takes a very long time and (for the duration I’ve waited for) it produces very inefficient designs with D errors in the millions and S estimates around values 1x1033
The first time I did this I developed a random 30% of the total possible combinations for one alternative and combined these in all possible combinations of two alternatives to produce choice sets which produced 28,227 choice sets. This took over 1.5 hours for Ngene to find one valid design.
I then took a random 10% of these choice sets (2822) and used it. This was faster, came across a valid design within one minute, but still has similarly large D errors and S estimates after about an hour running.
To me this seems a method that may eventually find a good design but would take an unreasonably long time and so does not seem a suitable solution, if the way I have set it up is accurate.

Option 3: change the syntax
Another option which I have tried is by only including the proc and ben attributes in the first alternative in the Ngene design, such that the syntax is:

Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/

U(alt2) = bword*word +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bcross*cross $

This works fairly nicely, there are no errors, it is fast and produces not totally unreasonable S estimates – around 500. I was surprised at how much the S estimate increased by removing the proc and ben attributes from alternative 2 however – in the original design the estimate is around 150.
I’m also not sure if this is ok and if it changes some assumptions about the way that efficiency is estimated or the designs selected such that it would result in misestimation come analysis time. Can you confirm if this design is ok for my purposes?

Option 4: remove the two attributes from the design in Ngene, and then manually add them to the design in something like R.
Again, this will work quite nicely in that the smaller Ngene design works quickly, produces good D errors and S estimates and I can add the proc and ben attributes afterwards using a counterbalanced or randomised allocation of levels across the choice sets to ensure each appears equally (or approximately equally). But again, I am not sure if this is problematic from an estimation point of view.
I understand there is a lot in this question, so apologise, but would be very grateful for any insight and suggestions.

Thanks
Laura

[Michiel Bliemer]

Section 8.5 of the Ngene manual deals with scenario variables in which attribute levels are the same within a choice task.

Below I changed the syntax to refer proc to the previous proc and ben to the previous ben by using proc[proc] and ben[ben], or by using constraints. As you will see, this creates designs with an infinite D-error because such a model cannot be estimated; when you fix an attribute in an unlabelled experiment, then the coefficient of this attribute cannot be estimated because the respondent never makes a trade-off between two different attribute levels.

Code: Select all

Design ? Using default swapping algorithm
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/

U(alt2) = bword*word + bproc*proc[proc] +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben[ben] + bcross*cross $

Code: Select all

Design ? Using modified Federov algorithm
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;alg = mfederov
;require:
alt1.proc = alt2.proc,
alt1.ben = alt2.ben
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/

U(alt2) = bword*word + bproc*proc +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben + bcross*cross $

In a labelled expeirment, you could include proc and ben only in one of the alternatives. In an unlabelled experiment, you can only estimate the model by creating interaction effects, for example:

... + bproc1 * proc[0,1] * word.dummy[0] + bproc2 * proc * word.dummy[1] + ...

So you will need to carefully consider the model that you want to estimate and which interactions to include as there are potentially many interactions with all dummy coded variables. It is a bit more tricky to create interactions between two dummy coded variables (i.e., with more than 2 levels) in Ngene.

Further, note that the last level in Ngene is always the reference level, so when you write bproc.dummy[0.586]*proc[0,1] then level 0 is associated with a utility of 0.586 and level 1 is associated with a utility of 0.

Michiel

[LauraWall]

Thanks Michiel, and thanks for the quick response

Section 8.5 of the Ngene manual deals with scenario variables in which attribute levels are the same within a choice task.
Thankyou, this is exactly the kind of set up what I want, and I had wondered if treating the proc and ben variables as essentially covariates was going to be the best way to go

I am surprised by how different the D error and S estimate are however between designs that include the ben and proc variables in this way vs don't include them (or include them as more standard attributes, varying within choice sets)

if I run the design without ben or proc the D error is around 0.2 and the S estimate around 70
When I then include just proc I would expect it to jump but not to S estimates of 10^30
And then, what I also don't understand is if I then include ben as well, it jumps but only to 10^33

Why is it that including one covariate (with only two levels) causes an increase in the estimated minimum sample size of 10^28 times as large, but then additionally including another with three levels only increases this sample size by around 1000 times?

I also don't understand why the S estimate is so low, around 70, for a design with 6 attributes, but then if I test what a design with just proc and ben would be if they varied within choice sets, then the S estimate is around 200

should not the number of attributes and levels have more of a relationship with the minimum sample size estimate ?

Laura

[Michiel Bliemer]

Any value like 10^30 means INFINITE, i.e. you cannot estimate the model as I explained in my email. You should NOT use the syntax that I sent as I explained, this was simply to show that this model CANNOT BE ESTIMATED. In order to estimate models with scenarios, you need (i) labelled alternatives, e.g. adding a no-choice or optout alternative, or (ii) create interactions.

There is no point discussing D-errors and S-estimates of a model that cannot be estimated, you should ignore any output as none of it makes sense, everything is infinite, so once you have formulated a model where the coefficients are identifiable I am happy to further assist.

Michiel

[LauraWall]

Hi Michiel

Thanks for following up

Ok, that makes more sense now why those values were occurring - I had forgot to mention the additional opt out question in my design which makes the proc and ben attributes make a bit more sense

In some ways my DCE could be considered as having a labelled, third, opt out option ...
In my DCE, we are interested in designing the best clinical trial from a patients perspective. In doing so we want to know which features would result in patients joining or not joining trials and this is captured in the DCE by this 'opt out' style question where participants are asked for each choice set, would you join this trial, yes or no.
However, because the trials we are examining generally have very low rates of participation, if we were to go with this design where no is a third option, then it is very likely that almost all respondents would select this option for most if not all choice sets and we would not get enough information. As such, we will force respondents to indicate which option they prefer (giving us information about relative preferences of attributes) and then once an option is selected as preferred respondents will be able to 'opt out' and indicate that they would or would not join that trial if presented with it in real life.

I had been intending on analysing and modelling these separately - first the preference data -- and then in a second analysis the join/opt out data
This second analysis is how I had been intending on making use of the proc and ben data

Would you consider this type of design as therefore having a labelled opt out alternative or not?

In regards to interactions, we do expect that there may be some interactions with the proc attribute, but it is difficult for us to know before running the study however which factors may interact - we were hoping the pilot study (which I am designing currently) would help inform this

it is unlikely the ben attribute would have any interactions - its inclusion was purely to see if it affected people's likelihood of joining the trial

Laura

[Michiel Bliemer]

Ok then please put the optout as an alternative, the code below will work:

Code: Select all

Design ? Using default swapping algorithm
;alts = alt1, alt2, optout
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/

U(alt2) = bword*word + bproc*proc[proc] +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben[ben] + bcross*cross $

Note that you will need to set priors such that they make sense with the optout, preferably from a pilot study. Also alt2 will need a constant, probably the same b1, and this constant will need a properly set prior. Without an appropriate prior for b1, the design will become inefficient and then it is best to use all zero priors for all attributes.

It is common to ask for both the unconditional (incl optout) and conditional (without optout) choice such that you obtain all information required. We generally first ask the unconditional choice and if people select alt1 or alt2 you do not need to ask the conditional choice, but that is a matter of preference.

Michiel

[LauraWall]

Great, thanks so much Michiel, I have asked lots of people and none have helped me pick up this was where I was going wrong, that I had forgotten to include a critical part of my design in my syntax

I feel so silly but glad that it all makes sense now

Great to know as well that this is a common approach

Laura