Dear all
I’m still an Ngene novice and hoped for some advice/help please regarding working out the number of choice tasks (S) for my DCE.
I intend to analyse data collected from a DCE using panel mixed multinomial logit model (MMNL).
I know that with Ngene, even though I will be analysing using panel mixed multinomial logit model (MMNL), I can still design choice tasks using a multinomial logit (MNL) model.
So using the MNL model for designing DCE, I have:
11 parameter coefficients i.e. K=11
J=3 so (J-1)=(3-1)=2
minimum number of choice tasks (S) = S >= K÷(J-1)
so S >= 11÷2 = minimum of 6 choice tasks for my DCE.
However, as I will be analysing collected data using a panel mixed multinomial logit model (MMNL) and I will be estimating a mean and standard deviation for each coefficient, this increases the number of parameters. Usually, this means doubling the minimum number of choice tasks required (S).
1). So I should double my minimum number of choice tasks to a minimum of 12 choice tasks for my DCE?
2). As we shouldn’t use the bare minimum number of choice tasks, does that mean that I should again double my minimum number of choice tasks (i.e. 12 x 2)?
3). Does that mean that I should be using 24 choice tasks for my DCE?
Apologies for the simple, silly questions.
Thanks,
Rakhee.
Number of choice tasks for DCE
Moderators: Andrew Collins, Michiel Bliemer, johnr
7 posts
• Page 1 of 1
Number of choice tasks for DCE
by rakhee » Thu Jan 17, 2019 11:48 am
- rakhee
- Posts: 15
- Joined: Thu Dec 20, 2018 8:52 am
Re: Number of choice tasks for DCE
by johnr » Fri Jan 18, 2019 8:22 am
Hi Rakhee
1. K in the equation is number of parameters being estimated, not number of attributes. So if you intend to estimate a MMNL model with random parameters with two moments, then each random parameter will require the estimation of two separate parameters. So yes, double the number of K in the calculation.
2. You can calculate this directly by treating K as the total number of parameters to be estimated. If all parameters are random, then K = 22. This as you suggest gives a minimum of 11 (rounded to 12) representing the very minimum necessary but not sufficient number you need mathematically to work the matrices in the background. Going higher than this amount is good from a conservative perspective (and probably publication - many reviewers don't understand this calculation and will probably reject a paper if you only have 12 tasks in total - even if they are the most informative tasks in terms of information content captured).
3. There is no answer to this question. It is a question of how comfortable you (and reviewers would be) with such a design. The answer is probably not very comfortable. A small number of tasks covers only a small number of design space (you don't have a lot of level combinations covered) which some reviewers will argue is bad (remember the calculation you cite is about matrices and inversion, not design space covered). I would argue whilst true, by selecting an efficient design, you are selecting the attribute level combinations that provide the most information from the respondent. Statistically, each choice tasks provides different amounts of information. Think of it this way, assume I was working on a problem that involved 1000 possible choice tasks, and I wanted to select only 10. If I could work out the amount of information each choice task provided, I could rank all 1000 and select only the first 10. Why do different choice tasks provide different degrees of information? Some attribute level combinations are more useful in working out preferences. If I have a choice task in which the levels for one alternative are much better than the levels of all other alternatives, everyone in the sample would be expected to choose this better alternative - I get no information about how people trade-off the levels. Contrast this to a choice task where every alternative is best on at least one level. I can now observe that some people select the low price alternative, whilst others the alternative that has say a lower travel time. By looking how the choices vary over the sample, I get information.
This is precisely what Michiel and I argued here:
Bliemer, M.C.J. and Rose, J.M. (2011) Experimental design influences on stated choice outputs: an empirical study in air travel choice, Transportation Research Part A, 45(1), 63-79.
In this paper, we had three designs
a) orthogonal with 108 choice tasks
b) an efficient design with 108 choice tasks
c) an efficient design with 18 choice tasks.
The take away of the study was that empirically, design c performed no worse than designs b and c in terms of standard errors. That is, going back to my example with 1000 choice tasks, each additional choice task provides diminishing returns in terms of information content. If I selected 11 instead of 10 choice tasks, the 11th provides slightly less than the 10th ranked task, which provides less than the 9th ranked task. By the time you get to the 100th task, you are probably (this will be situational specific) not adding much more information than if you had 99 tasks.
The problem however as I said is not mathematics. Its reviewers. Sometimes you go larger to cut-off a potential reviewer comment (in my experience, there are reviewers who you can argue all day long with, present as much evidence as you like, they will still not listen as they have an ideological view of the world that will not budge). So is 24 enough - mathematically - likely yes (again remember this number is number is necessary but not sufficient). From your literature (from whom reviewers will be drawn), I cannot answer.
John
1. K in the equation is number of parameters being estimated, not number of attributes. So if you intend to estimate a MMNL model with random parameters with two moments, then each random parameter will require the estimation of two separate parameters. So yes, double the number of K in the calculation.
2. You can calculate this directly by treating K as the total number of parameters to be estimated. If all parameters are random, then K = 22. This as you suggest gives a minimum of 11 (rounded to 12) representing the very minimum necessary but not sufficient number you need mathematically to work the matrices in the background. Going higher than this amount is good from a conservative perspective (and probably publication - many reviewers don't understand this calculation and will probably reject a paper if you only have 12 tasks in total - even if they are the most informative tasks in terms of information content captured).
3. There is no answer to this question. It is a question of how comfortable you (and reviewers would be) with such a design. The answer is probably not very comfortable. A small number of tasks covers only a small number of design space (you don't have a lot of level combinations covered) which some reviewers will argue is bad (remember the calculation you cite is about matrices and inversion, not design space covered). I would argue whilst true, by selecting an efficient design, you are selecting the attribute level combinations that provide the most information from the respondent. Statistically, each choice tasks provides different amounts of information. Think of it this way, assume I was working on a problem that involved 1000 possible choice tasks, and I wanted to select only 10. If I could work out the amount of information each choice task provided, I could rank all 1000 and select only the first 10. Why do different choice tasks provide different degrees of information? Some attribute level combinations are more useful in working out preferences. If I have a choice task in which the levels for one alternative are much better than the levels of all other alternatives, everyone in the sample would be expected to choose this better alternative - I get no information about how people trade-off the levels. Contrast this to a choice task where every alternative is best on at least one level. I can now observe that some people select the low price alternative, whilst others the alternative that has say a lower travel time. By looking how the choices vary over the sample, I get information.
This is precisely what Michiel and I argued here:
Bliemer, M.C.J. and Rose, J.M. (2011) Experimental design influences on stated choice outputs: an empirical study in air travel choice, Transportation Research Part A, 45(1), 63-79.
In this paper, we had three designs
a) orthogonal with 108 choice tasks
b) an efficient design with 108 choice tasks
c) an efficient design with 18 choice tasks.
The take away of the study was that empirically, design c performed no worse than designs b and c in terms of standard errors. That is, going back to my example with 1000 choice tasks, each additional choice task provides diminishing returns in terms of information content. If I selected 11 instead of 10 choice tasks, the 11th provides slightly less than the 10th ranked task, which provides less than the 9th ranked task. By the time you get to the 100th task, you are probably (this will be situational specific) not adding much more information than if you had 99 tasks.
The problem however as I said is not mathematics. Its reviewers. Sometimes you go larger to cut-off a potential reviewer comment (in my experience, there are reviewers who you can argue all day long with, present as much evidence as you like, they will still not listen as they have an ideological view of the world that will not budge). So is 24 enough - mathematically - likely yes (again remember this number is number is necessary but not sufficient). From your literature (from whom reviewers will be drawn), I cannot answer.
John
- johnr
- Posts: 171
- Joined: Fri Mar 13, 2009 7:15 am
Re: Number of choice tasks for DCE
by rakhee » Fri Jan 18, 2019 10:48 am
Hi John
Thank you so much for that great explanation.
I did forget to include the following information in my initial post:
a). I intended to use an efficient design in Ngene, and not an orthogonal design.
b). My sample size is likely going to be around 800.
After reading your explanations, my thought process to get to my number of choice tasks required when using an efficient design in Ngene is as follows (apologies for the repetition):
1). Yes, when specifying K, this is the number of parameters I will be estimating in my utility equation. K is not the number of attributes in my DCE.
2). When estimating using MNL model I will have 11 parameter coefficents (i.e. K=11). Here, the minimum number of choice tasks (S) is 6 (i.e. 5.5 rounded up).
3). When estimating using MMNL model, each parameter from the previous MNL model will have a mean and std dev hence doubling the number of coefficients. So with the MMNL model I will have 22 parameter coefficients therefore K=22. Here, the minimum number of choice tasks (S) is 12 (i.e. 11 rounded up).
Note. My understanding is that it is better to design for more random parameters rather than too little.i.e. if design for K=22 then I am reasonably confident to analyse for K=22. Even though in reality, during analysis when using MMNL model I may choose not to make all my parameter coefficients random i.e. K will be <22.
4). Based on your answers (i.e. reviewers don't like seeing papers using the lower bound minimum number of choice tasks, and that I want to capture enough variation in my data), it is advisable to then double my lower bound minimum number of choice tasks (S=12) i.e. 2 x S = 2 x 12 choice tasks = 24 choice tasks.
5). So my decision is to use 24 choice tasks, in either 4 blocks of 6 or 3 blocks of 8, to create my efficient design in Ngene for my DCE choice tasks.
Hopefully these are my final 2 questions for this topic and you don't mind answering them please:
Question 1: Do you see any flaws/red flags in my thought process above, which has led to my decision to use 24 choice tasks, in either 4 blocks of 6 or 3 blocks of 8, when creating my efficient design in Ngene?
Question 2: Have I missed anything that would allow me to use a lower number of choice tasks when using an efficient design in Ngene for my DCE choice tasks?
Many thanks,
Rakhee.
Thank you so much for that great explanation.
I did forget to include the following information in my initial post:
a). I intended to use an efficient design in Ngene, and not an orthogonal design.
b). My sample size is likely going to be around 800.
After reading your explanations, my thought process to get to my number of choice tasks required when using an efficient design in Ngene is as follows (apologies for the repetition):
1). Yes, when specifying K, this is the number of parameters I will be estimating in my utility equation. K is not the number of attributes in my DCE.
2). When estimating using MNL model I will have 11 parameter coefficents (i.e. K=11). Here, the minimum number of choice tasks (S) is 6 (i.e. 5.5 rounded up).
3). When estimating using MMNL model, each parameter from the previous MNL model will have a mean and std dev hence doubling the number of coefficients. So with the MMNL model I will have 22 parameter coefficients therefore K=22. Here, the minimum number of choice tasks (S) is 12 (i.e. 11 rounded up).
Note. My understanding is that it is better to design for more random parameters rather than too little.i.e. if design for K=22 then I am reasonably confident to analyse for K=22. Even though in reality, during analysis when using MMNL model I may choose not to make all my parameter coefficients random i.e. K will be <22.
4). Based on your answers (i.e. reviewers don't like seeing papers using the lower bound minimum number of choice tasks, and that I want to capture enough variation in my data), it is advisable to then double my lower bound minimum number of choice tasks (S=12) i.e. 2 x S = 2 x 12 choice tasks = 24 choice tasks.
5). So my decision is to use 24 choice tasks, in either 4 blocks of 6 or 3 blocks of 8, to create my efficient design in Ngene for my DCE choice tasks.
Hopefully these are my final 2 questions for this topic and you don't mind answering them please:
Question 1: Do you see any flaws/red flags in my thought process above, which has led to my decision to use 24 choice tasks, in either 4 blocks of 6 or 3 blocks of 8, when creating my efficient design in Ngene?
Question 2: Have I missed anything that would allow me to use a lower number of choice tasks when using an efficient design in Ngene for my DCE choice tasks?
Many thanks,
Rakhee.
- rakhee
- Posts: 15
- Joined: Thu Dec 20, 2018 8:52 am
Re: Number of choice tasks for DCE
by johnr » Fri Jan 18, 2019 1:08 pm
Hi Rakhee
I think you have the idea. I personally can't see any flaw in your logic in terms of design generation, but that said, I suggest a little caution when modelling with large numbers of random parameters (11 is a lot). You can quickly run into identification issues (again, I'm talking modelling, not design). So to be explicit in answering your questions
1) From what you have written, this seems fine and logical, however I cannot answer how knowledgeable reviewers are in your field and whether or not they will understand (we run into reviewers with little understanding all the time - I'm dealing with two papers currently where one reviewer for each (different reviewers I hope) don't appear to have even a basic understanding of choice modelling and no matter what we reply, they just don't get it). I would suggest however that you optimise the design for MNL and test how it would work for a MMNL model specification (you can do this in Ngene or run simulations yourself). At least that way, you can tackle any potential reviewer head on and say you have done X, and tested the design using (simulations) and found it worked!
2) Did you consider ASCs in your d.f. calculation? Perhaps testing how the design will perform on MMNL as extra layer of protection. Also, how about a conducting a small pilot and performing tests. Lastly, if you can tell me what is happening to the Australian cricket team would appreciate it... this is becoming depressing!
John
I think you have the idea. I personally can't see any flaw in your logic in terms of design generation, but that said, I suggest a little caution when modelling with large numbers of random parameters (11 is a lot). You can quickly run into identification issues (again, I'm talking modelling, not design). So to be explicit in answering your questions
1) From what you have written, this seems fine and logical, however I cannot answer how knowledgeable reviewers are in your field and whether or not they will understand (we run into reviewers with little understanding all the time - I'm dealing with two papers currently where one reviewer for each (different reviewers I hope) don't appear to have even a basic understanding of choice modelling and no matter what we reply, they just don't get it). I would suggest however that you optimise the design for MNL and test how it would work for a MMNL model specification (you can do this in Ngene or run simulations yourself). At least that way, you can tackle any potential reviewer head on and say you have done X, and tested the design using (simulations) and found it worked!
2) Did you consider ASCs in your d.f. calculation? Perhaps testing how the design will perform on MMNL as extra layer of protection. Also, how about a conducting a small pilot and performing tests. Lastly, if you can tell me what is happening to the Australian cricket team would appreciate it... this is becoming depressing!
John
- johnr
- Posts: 171
- Joined: Fri Mar 13, 2009 7:15 am
Re: Number of choice tasks for DCE
by rakhee » Fri Jan 18, 2019 2:39 pm
Hi John,
Again, thank you very much for looking through my query and providing guidance.
I hear what you are advising regarding modelling 11 random parameters. I don't really intend on modelling that many but like I said before...my understanding after conversations with Michiel is that design should account for more parameters that you may model for, rather than the other way round.
The other piece of advice I'm hearing from you here, is maybe I could be a bit more selective when identifying which parameters I would make random during modelling (which will in turn change my number of K), and then do the choice task calculation again for the efficient design. This may reduce the number of choice tasks required.
1). Yes, when actually carrying out the efficient design in Ngene for the main study, I intend to optimise the design for MNL model, and also evaluate for the MMNL model. Trying to remember from my Ngene folder, that means I optimise for MNL model but I should also include the utility equation for my rp panel model and tick the RP Panel box in the Properties tab to evaluate the MMNL model...I think that's what you mean, otherwise I don't know how to do what you've suggested.
I'll look through my Ngene course folder to remind myself how to do this again anyway. As far as I know, I won't be able to use this strategy for my pilot study design as I'll be using zero priors (as the literature doesn't give me any idea about size or sign to use for parameter priors during design for pilot study) i.e. my pilot study will only optimise on MNL model for design.
2). Yes thanks, I did account for the ASCs in the d.f calculation..I just counted each of the ASCs as a parameter coefficient.
My questions regarding this topic relate to my choice task design for the pilot study I intend to conduct first. Just a heads up for you
...I'll be using a pivot design and constraints...things I'm not really familiar with so I will most probably be posting another topic later on seeking advice when I've tried creating my design syntax for the pilot study design.
Thanks for providing advice that takes into account how reviewers may view my work. I do hope that down the track I am able consolidate and explain my methodology well enough.
I am an avid cricket fan, a New Zealander, and more that happy to talk cricket anytime...but maybe better not to discuss the Australian cricket team on a public forum
Again, thank very much for looking at my query and providing advice.
Rakhee.
Again, thank you very much for looking through my query and providing guidance.
I hear what you are advising regarding modelling 11 random parameters. I don't really intend on modelling that many but like I said before...my understanding after conversations with Michiel is that design should account for more parameters that you may model for, rather than the other way round.
The other piece of advice I'm hearing from you here, is maybe I could be a bit more selective when identifying which parameters I would make random during modelling (which will in turn change my number of K), and then do the choice task calculation again for the efficient design. This may reduce the number of choice tasks required.
1). Yes, when actually carrying out the efficient design in Ngene for the main study, I intend to optimise the design for MNL model, and also evaluate for the MMNL model. Trying to remember from my Ngene folder, that means I optimise for MNL model but I should also include the utility equation for my rp panel model and tick the RP Panel box in the Properties tab to evaluate the MMNL model...I think that's what you mean, otherwise I don't know how to do what you've suggested.
I'll look through my Ngene course folder to remind myself how to do this again anyway. As far as I know, I won't be able to use this strategy for my pilot study design as I'll be using zero priors (as the literature doesn't give me any idea about size or sign to use for parameter priors during design for pilot study) i.e. my pilot study will only optimise on MNL model for design.
2). Yes thanks, I did account for the ASCs in the d.f calculation..I just counted each of the ASCs as a parameter coefficient.
My questions regarding this topic relate to my choice task design for the pilot study I intend to conduct first. Just a heads up for you
...I'll be using a pivot design and constraints...things I'm not really familiar with so I will most probably be posting another topic later on seeking advice when I've tried creating my design syntax for the pilot study design. Thanks for providing advice that takes into account how reviewers may view my work. I do hope that down the track I am able consolidate and explain my methodology well enough.
I am an avid cricket fan, a New Zealander, and more that happy to talk cricket anytime...but maybe better not to discuss the Australian cricket team on a public forum
Again, thank very much for looking at my query and providing advice.
Rakhee.
- rakhee
- Posts: 15
- Joined: Thu Dec 20, 2018 8:52 am
Re: Number of choice tasks for DCE
by johnr » Fri Jan 18, 2019 3:59 pm
Hi Rakhee
Yep, you appear to have understood everything discussed. Always happy to answer questions. That is what we are here for, although I note that every time you reply here, Australia loose a wicket. You keep this up, and we may have to ban you from this forum.
John
Yep, you appear to have understood everything discussed. Always happy to answer questions. That is what we are here for, although I note that every time you reply here, Australia loose a wicket. You keep this up, and we may have to ban you from this forum.
John
- johnr
- Posts: 171
- Joined: Fri Mar 13, 2009 7:15 am
Re: Number of choice tasks for DCE
by rakhee » Mon Jan 21, 2019 8:26 am
Hi John,
Told you that no good can come from discussing the Australian cricket team on a public forum...there are always consequences
Anyway, I wouldn't be writing them off...they've got good talent and are in a re-building phase.
Thanks again for the advice and guidance John.
Rakhee.
Told you that no good can come from discussing the Australian cricket team on a public forum...there are always consequences
Anyway, I wouldn't be writing them off...they've got good talent and are in a re-building phase.
Thanks again for the advice and guidance John.
Rakhee.
- rakhee
- Posts: 15
- Joined: Thu Dec 20, 2018 8:52 am
7 posts
• Page 1 of 1
Return to Choice experiments - Ngene
Who is online
Users browsing this forum: No registered users and 26 guests