fixed levels within choice sets for some attributes
Posted: Tue Nov 10, 2020 2:33 pm
Dear NGene team
I am working on a design with 8 attributes and two alternatives, see below (number of rows and blocks TBD based on efficiency and S estimates from design to match practical recruitment constraints):
Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/
U(alt2) = bword*word + bproc*proc +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben + bcross*cross $
I want the procedure (proc) and benefit (ben) attributes to have a fixed level within choice sets but this level vary across choice sets
I have come across a number of possible ways to achieve this but each have issues that cause me concern and I would like some advice over the specific issues within each and also advice over which approach is most appropriate
Option1: create the design in Ngene as if the levels do change for each alternative (syntax as above) and then evaluate the design but with the require property by adding the following lines before ;model:
;eval= Design.csv / Design.ngd
;require:
alt1.proc = alt2.proc,
alt1.ben = alt2.ben
This appears to work (using either the Ngene saved design, or the output from the design saved into a csv and then uploaded) in the sense that I do not get any errors – however the D error, S estimate and all other information (including the design itself) is exactly the same whether I have or do not have the require argument.
I therefore assume that it is not doing what it should, as surely requiring this should change something about the design right?
Option 2: use an availability / partial profiles design by developing the possible choice sets in another program such as R (with the levels for these attributes fixed for each choice set), and then uploading these possible choice sets and selecting an appropriate design for them by adding the following line before ;model:
;alg = mfederov(candidates = explicitPartialProfiles.csv)
Again, this works in the sense that it does not produce an error but it takes a very long time and (for the duration I’ve waited for) it produces very inefficient designs with D errors in the millions and S estimates around values 1x1033
The first time I did this I developed a random 30% of the total possible combinations for one alternative and combined these in all possible combinations of two alternatives to produce choice sets which produced 28,227 choice sets. This took over 1.5 hours for Ngene to find one valid design.
I then took a random 10% of these choice sets (2822) and used it. This was faster, came across a valid design within one minute, but still has similarly large D errors and S estimates after about an hour running.
To me this seems a method that may eventually find a good design but would take an unreasonably long time and so does not seem a suitable solution, if the way I have set it up is accurate.
Option 3: change the syntax
Another option which I have tried is by only including the proc and ben attributes in the first alternative in the Ngene design, such that the syntax is:
Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/
U(alt2) = bword*word +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bcross*cross $
This works fairly nicely, there are no errors, it is fast and produces not totally unreasonable S estimates – around 500. I was surprised at how much the S estimate increased by removing the proc and ben attributes from alternative 2 however – in the original design the estimate is around 150.
I’m also not sure if this is ok and if it changes some assumptions about the way that efficiency is estimated or the designs selected such that it would result in misestimation come analysis time. Can you confirm if this design is ok for my purposes?
Option 4: remove the two attributes from the design in Ngene, and then manually add them to the design in something like R.
Again, this will work quite nicely in that the smaller Ngene design works quickly, produces good D errors and S estimates and I can add the proc and ben attributes afterwards using a counterbalanced or randomised allocation of levels across the choice sets to ensure each appears equally (or approximately equally). But again, I am not sure if this is problematic from an estimation point of view.
I understand there is a lot in this question, so apologise, but would be very grateful for any insight and suggestions.
Thanks
Laura
I am working on a design with 8 attributes and two alternatives, see below (number of rows and blocks TBD based on efficiency and S estimates from design to match practical recruitment constraints):
Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/
U(alt2) = bword*word + bproc*proc +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bben*ben + bcross*cross $
I want the procedure (proc) and benefit (ben) attributes to have a fixed level within choice sets but this level vary across choice sets
I have come across a number of possible ways to achieve this but each have issues that cause me concern and I would like some advice over the specific issues within each and also advice over which approach is most appropriate
Option1: create the design in Ngene as if the levels do change for each alternative (syntax as above) and then evaluate the design but with the require property by adding the following lines before ;model:
;eval= Design.csv / Design.ngd
;require:
alt1.proc = alt2.proc,
alt1.ben = alt2.ben
This appears to work (using either the Ngene saved design, or the output from the design saved into a csv and then uploaded) in the sense that I do not get any errors – however the D error, S estimate and all other information (including the design itself) is exactly the same whether I have or do not have the require argument.
I therefore assume that it is not doing what it should, as surely requiring this should change something about the design right?
Option 2: use an availability / partial profiles design by developing the possible choice sets in another program such as R (with the levels for these attributes fixed for each choice set), and then uploading these possible choice sets and selecting an appropriate design for them by adding the following line before ;model:
;alg = mfederov(candidates = explicitPartialProfiles.csv)
Again, this works in the sense that it does not produce an error but it takes a very long time and (for the duration I’ve waited for) it produces very inefficient designs with D errors in the millions and S estimates around values 1x1033
The first time I did this I developed a random 30% of the total possible combinations for one alternative and combined these in all possible combinations of two alternatives to produce choice sets which produced 28,227 choice sets. This took over 1.5 hours for Ngene to find one valid design.
I then took a random 10% of these choice sets (2822) and used it. This was faster, came across a valid design within one minute, but still has similarly large D errors and S estimates after about an hour running.
To me this seems a method that may eventually find a good design but would take an unreasonably long time and so does not seem a suitable solution, if the way I have set it up is accurate.
Option 3: change the syntax
Another option which I have tried is by only including the proc and ben attributes in the first alternative in the Ngene design, such that the syntax is:
Design
;alts = alt1, alt2
;rows = 36
;block = 3
;eff = (mnl,d)
;model:
U(alt1) = b1 + bword.dummy[0.117|0.206]*word[0,1,2] +
bproc.dummy[0.586]*proc[0,1] +
bsham.dummy[0.585]*sham[0,1] +
barms.dummy[0.486|0.485]*arms[0,1,2] +
btime.dummy[0.648|0.899]*time[0,1,2] +
bcomm.dummy[0.201|0.255]*comm[0,1,2] +
bben.dummy[0.101|0.106]*ben[0,1,2] +
bcross.dummy[0.302]*cross[0,1]/
U(alt2) = bword*word +
bsham*sham + barms*arms + btime*time +
bcomm*comm + bcross*cross $
This works fairly nicely, there are no errors, it is fast and produces not totally unreasonable S estimates – around 500. I was surprised at how much the S estimate increased by removing the proc and ben attributes from alternative 2 however – in the original design the estimate is around 150.
I’m also not sure if this is ok and if it changes some assumptions about the way that efficiency is estimated or the designs selected such that it would result in misestimation come analysis time. Can you confirm if this design is ok for my purposes?
Option 4: remove the two attributes from the design in Ngene, and then manually add them to the design in something like R.
Again, this will work quite nicely in that the smaller Ngene design works quickly, produces good D errors and S estimates and I can add the proc and ben attributes afterwards using a counterbalanced or randomised allocation of levels across the choice sets to ensure each appears equally (or approximately equally). But again, I am not sure if this is problematic from an estimation point of view.
I understand there is a lot in this question, so apologise, but would be very grateful for any insight and suggestions.
Thanks
Laura