Holdout task

This forum is for posts covering broader stated choice experimental design issues.

Moderators: Andrew Collins, Michiel Bliemer, johnr

Holdout task

Postby bye1830 » Tue Oct 11, 2022 4:23 pm

Dear Ngene Team,

Thank you for all your help and support. I really appreciate it.

I have another question, but it is about a different, more general topic. So, let me upload this posting here.

I have found some previous studies in which their choice experiments included a holdout task, which is also called a fixed, or a control choice task. It seems that this holdout task can be used to test predictive validity of a calibrated model.

I wonder if you have any recommendations on the design of the holdout task. For example,

1) How many holdout tasks are appropriate? (e.g., one vs. two?)
2) Can I select any attribute levels for a holdout task?
3) Can a holdout task be located anywhere in a series of choice tasks? Or, is there a more proper location (e.g., in the middle or at the very end)?

Thank you very much again. I'd really appreciate it if you could give me any inputs.

Best regards,

yb
bye1830
 
Posts: 11
Joined: Wed Jun 22, 2022 3:19 pm

Re: Holdout task

Postby Michiel Bliemer » Tue Oct 11, 2022 9:42 pm

When testing the predictive validity of any model, it is common to use 80% of your data for estimation/calibration, and 20% for testing/predicting. Note that the holdout sample does not need to consist of a specific choice task(s). Rather, you can do the following:

1. Randomly select 80% of your choice tasks and use these to estimate the parameters
2. Apply the estimated/calilbrated choice model to 20% of the remaining choice tasks to test the predictive capability
3. Repeat the process (e.g. 10 times)

This means that you are using all choice tasks in model estimation and in validation, but not at the same time.

Michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm

Re: Holdout task

Postby bye1830 » Sat Oct 15, 2022 4:39 pm

Dear Michiel,

Thank you so much again. This is the knowledge that I didn't know before. Very helpful.

1. Could you please share with me a reference of this method, if you know any? I might need to cite one when I'm writing a paper.

2. If we do random selections of choice tasks for calibration and validation each, and repeat this process 10 times or so, how to pick a subset of the choice tasks for a "final" calibrated model (i.e., the estimated model to be reported in a paper)? Are there any guidelines?

Thank you for your time and help.

Best regards,

yb
bye1830
 
Posts: 11
Joined: Wed Jun 22, 2022 3:19 pm

Re: Holdout task

Postby Michiel Bliemer » Sat Oct 15, 2022 5:14 pm

The 80:20 rule is just a rule of thumb that I have heard from other data analytics disciplines such as machine learning, this is where they often do forecasting. I do not know the original source.

In choice modelling, it is common to use all the data for model estimation. Choice modellers do not often evaluate prediction power, but when doing so they can follow the same strategy as in other data analytics methods using the 80:20 rule.

Maybe someone else has done such analyses, I have not, so I am not sure what the convention is in calibrating choice models in this way.

michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm

Re: Holdout task

Postby bye1830 » Tue Oct 18, 2022 5:06 pm

Oh I see. Thank you Michiel for your reply. That's very helpful. I've just Googled the 80:20 rule; it looks like a split ratio commonly used between training and testing sets in machine learning. I'll search this further.

In case someone else is looking for information on holdout tasks from this posting, let me just share some articles that I found. Seems that 10~20% of the tasks have been used for validation in these choice experiment-based studies.


Regards,

yb
bye1830
 
Posts: 11
Joined: Wed Jun 22, 2022 3:19 pm

Re: Holdout task

Postby Michiel Bliemer » Tue Oct 18, 2022 5:26 pm

Thank you for sharing, that is very considerate of you and I hope this helps others.

Michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm


Return to Choice experiments - general

Who is online

Users browsing this forum: No registered users and 3 guests

cron