Holdout task

[bye1830]

Dear Ngene Team,

Thank you for all your help and support. I really appreciate it.

I have another question, but it is about a different, more general topic. So, let me upload this posting here.

I have found some previous studies in which their choice experiments included a holdout task, which is also called a fixed, or a control choice task. It seems that this holdout task can be used to test predictive validity of a calibrated model.

I wonder if you have any recommendations on the design of the holdout task. For example,

1) How many holdout tasks are appropriate? (e.g., one vs. two?)
2) Can I select any attribute levels for a holdout task?
3) Can a holdout task be located anywhere in a series of choice tasks? Or, is there a more proper location (e.g., in the middle or at the very end)?

Thank you very much again. I'd really appreciate it if you could give me any inputs.

Best regards,

yb

[Michiel Bliemer]

When testing the predictive validity of any model, it is common to use 80% of your data for estimation/calibration, and 20% for testing/predicting. Note that the holdout sample does not need to consist of a specific choice task(s). Rather, you can do the following:

1. Randomly select 80% of your choice tasks and use these to estimate the parameters
2. Apply the estimated/calilbrated choice model to 20% of the remaining choice tasks to test the predictive capability
3. Repeat the process (e.g. 10 times)

This means that you are using all choice tasks in model estimation and in validation, but not at the same time.

Michiel

[bye1830]

Dear Michiel,

Thank you so much again. This is the knowledge that I didn't know before. Very helpful.

1. Could you please share with me a reference of this method, if you know any? I might need to cite one when I'm writing a paper.

2. If we do random selections of choice tasks for calibration and validation each, and repeat this process 10 times or so, how to pick a subset of the choice tasks for a "final" calibrated model (i.e., the estimated model to be reported in a paper)? Are there any guidelines?

Thank you for your time and help.

Best regards,

yb

[Michiel Bliemer]

The 80:20 rule is just a rule of thumb that I have heard from other data analytics disciplines such as machine learning, this is where they often do forecasting. I do not know the original source.

In choice modelling, it is common to use all the data for model estimation. Choice modellers do not often evaluate prediction power, but when doing so they can follow the same strategy as in other data analytics methods using the 80:20 rule.

Maybe someone else has done such analyses, I have not, so I am not sure what the convention is in calibrating choice models in this way.

michiel

[bye1830]

Oh I see. Thank you Michiel for your reply. That's very helpful. I've just Googled the 80:20 rule; it looks like a split ratio commonly used between training and testing sets in machine learning. I'll search this further.

In case someone else is looking for information on holdout tasks from this posting, let me just share some articles that I found. Seems that 10~20% of the tasks have been used for validation in these choice experiment-based studies.

• van Rijnsoever et al. (2013) - 10 tasks (including 1 control choice task)
  http://dx.doi.org/10.1016/j.trd.2013.01.005
  
• Apostolakis et al (2018) - 10 tasks (including 2 holdouts)
  https://doi.org/10.1016/j.jbef.2018.01.001
  
• Anderhofstadt & Spinler (2020) - 12 tasks (including 2 fixed tasks)
  https://doi.org/10.1016/j.trd.2020.102232
  
• Brazell et al. (2006) - 14 tasks (12 for calibration, 2 for validation)
  https://doi.org/10.1007/s11002-006-7943-8
  
• Hinz et al. (2015) - 14 tasks (12 for estimation, 2 for the holdouts)
  https://doi.org/10.1007/s11573-015-0765-5

Regards,

yb

[Michiel Bliemer]

Thank you for sharing, that is very considerate of you and I hope this helps others.

Michiel