No valid design has been found after 1000 evaluations!

[Reihaneh]

Dear Ngene team,

I am in the process of designing a survey to investigate the influence of residential-based policies on bike, auto, and transit pass ownership. For each scenario, I will ask respondents whether they plan to decrease, maintain, or increase their current number of bike, auto, or transit passes.

I have chosen to create three separate survey designs, each tailored to bike, auto, and transit policies. These designs will be combined into a single survey. I consider 'maintaining' as the reference utility and will evaluate each policy in terms of its effect on increasing or decreasing ownership, resulting in the same magnitude but with different signs for each prior. I also added constraints to have the same scenarios for both increasing and decreasing ultimately.

The design for the bike survey is as follows. However, I am encountering an error when I attempt to run it.
The error is [Warning: No valid design has been found after 1000 evaluations. There may be a problem with the specification of the design. A common problem is that the choice probabilities are too extreme (close to 1 and 0), perhaps because some or all of the prior values are too large. Also, it is generally a good idea to start with a simple design (MNL, non-Bayesian), and then add complexity. If you press stop, a design will be reported, which may assist in diagnosing the problem.]

Could you please help me identify any incorrect assumptions in this design and guide me on how to handle this type of survey design?
Thank you so much in advance,
Reihaneh



Design
;alts = decrease, maintain, increase
;rows = 12
;block = 2
;eff=(mnl,d)

;cond:
if(decrease.FreeBikeShareSt=0,increase.FreeBikeShareSt=0),
if(decrease.FreeBikeShareSt=1,increase.FreeBikeShareSt=1),
if(decrease.FreeBikeShareM=0,increase.FreeBikeShareM=0),
if(decrease.FreeBikeShareM=60,increase.FreeBikeShareM=60),
if(decrease.FreeBikeShareM=120,increase.FreeBikeShareM=120),
if(decrease.FreeBikePark=0,increase.FreeBikePark=0),
if(decrease.FreeBikePark=1,increase.FreeBikePark=1),
if(decrease.FreeEbikeSt=0,increase.FreeEbikeSt=0),
if(decrease.FreeEbikeSt=1,increase.FreeEbikeSt=1),
if(decrease.FreeEbikeVouch=0,increase.FreeEbikeVouch=0),
if(decrease.FreeEbikeVouch=10,increase.FreeEbikeVouch=10),
if(decrease.FreeEbikeVouch=30,increase.FreeEbikeVouch=30),
if(decrease.BikeRepairSt=0,increase.BikeRepairSt=0),
if(decrease.BikeRepairSt=1,increase.BikeRepairSt=1),
if(decrease.FreeBikeTunning=0,increase.FreeBikeTunning=0),
if(decrease.FreeBikeTunning=1,increase.FreeBikeTunning=1),
if(decrease.BikableAceessChildCare=0,increase.BikableAceessChildCare=0),
if(decrease.BikableAceessChildCare=1,increase.BikableAceessChildCare=1),
if(decrease.BikableAceessSchool=0,increase.BikableAceessSchool=0),
if(decrease.BikableAceessSchool=1,increase.BikableAceessSchool=1),
if(decrease.BikableAceessGrocery=0,increase.BikableAceessGrocery=0),
if(decrease.BikableAceessGrocery=1,increase.BikableAceessGrocery=1),
if(decrease.BikableAceessHospital=0,increase.BikableAceessHospital=0),
if(decrease.BikableAceessHospital=1,increase.BikableAceessHospital=1),
if(decrease.BikableAceessRestaurant=0,increase.BikableAceessRestaurant=0),
if(decrease.BikableAceessGym=0,increase.BikableAceessGym=0),
if(decrease.BikableAceessGym=1,increase.BikableAceessGym=1),
if(decrease.BikableAceessCommunity=0,increase.BikableAceessCommunity=0),
if(decrease.BikableAceessCommunity=1,increase.BikableAceessCommunity=1),
if(decrease.BikableAceessBank=0,increase.BikableAceessBank=0),
if(decrease.BikableAceessBank=1,increase.BikableAceessBank=1),
if(decrease.BikableAceessPetCare=0,increase.BikableAceessPetCare=0),
if(decrease.BikableAceessPetCare=1,increase.BikableAceessPetCare=1),
if(decrease.TelecommutingSpace=0,increase.TelecommutingSpace=0),
if(decrease.TelecommutingSpace=1,increase.TelecommutingSpace=1),
if(decrease.ShuttleService=0,increase.ShuttleService=0),
if(decrease.ShuttleService=1,increase.ShuttleService=1)




;model:
U(decrease)= b_bikeshareSt[(u,0.2,0.5)] * FreeBikeShareSt[0,1] +
b_bikeshareM[(u,0.001,0.03)] * FreeBikeShareM[0,60,120] +
b_bikePark[(u,-0.5,-0.2)] * FreeBikePark[0,1] +
b_bikeElecSt[(u,0.27,0.625)] * FreeEbikeSt[0,1] +
b_bikeElecVo[(u,-0.05,-0.02)] * FreeEbikeVouch[0,10,30] +
b_bikeRepSt[(u,-0.21,-0.09)] * BikeRepairSt[0,1] +
b_bikeRepTu[(u,-1.5,-0.5)] * FreeBikeTunning[0,1] +
b_AccChildcare[(u,-0.32,-0.08)]* BikableAceessChildCare[0,1] +
b_AccSchool[(u,-0.32,-0.08)] * BikableAceessSchool[0,1] +
b_AccGrocery[(u,-0.5,-0.3)] * BikableAceessGrocery[0,1] +
b_AccHospital[(u,-0.32,-0.08)] * BikableAceessHospital[0,1] +
b_AccRestaurant[(u,-0.5,-0.1)] * BikableAceessRestaurant[0,1] +
b_AccGym[(u,-0.42,-0.03)] * BikableAceessGym[0,1] +
b_AccCommunity[(u,-0.25,-0.03)]* BikableAceessCommunity[0,1] +
b_AccBank[(u,-0.5,-0.3)] * BikableAceessBank[0,1] +
b_AccPetCare[(u,-0.32,-0.08)] * BikableAceessPetCare[0,1] +
b_Telecommute[(u,0.2,0.5)] * TelecommutingSpace[0,1] +
b_ShuttleServ[(u,0.1,0.6)] * ShuttleService[0,1]

/


U(maintain)= A[(u,-0.5,0.5)]
/

U(increase)= b_bikeshareSt[(u,-0.5,-0.2)] * FreeBikeShareSt[0,1] +
b_bikeshareM[(u,-0.001,-0.03)] * FreeBikeShareM[0,60,120] +
b_bikePark[(u,0.2,0.5)] * FreeBikePark[0,1] +
b_bikeElecSt[(u,-0.625,-0.27)] * FreeEbikeSt[0,1] +
b_bikeElecVo[(u,0.02,0.05)] * FreeEbikeVouch[0,10,30] +
b_bikeRepSt[(u,0.09,0.21)] * BikeRepairSt[0,1] +
b_bikeRepTu[(u,0.5,1.5)] * FreeBikeTunning[0,1] +
b_AccChildcare[(u,0.08,0.32)] * BikableAceessChildCare[0,1] +
b_AccSchool[(u,0.08,0.32)] * BikableAceessSchool[0,1] +
b_AccGrocery[(u,0.3,0.5)] * BikableAceessGrocery[0,1] +
b_AccHospital[(u,0.08,0.32)] * BikableAceessHospital[0,1] +
b_AccRestaurant[(u,0.1,0.5)] * BikableAceessRestaurant[0,1] +
b_AccGym[(u,0.03,0.42)] * BikableAceessGym[0,1] +
b_AccCommunity[(u,0.03,0.25)] * BikableAceessCommunity[0,1] +
b_AccBank[(u,0.3,0.5)] * BikableAceessBank[0,1] +
b_AccPetCare[(u,0.08,0.32)] * BikableAceessPetCare[0,1] +
b_Telecommute[(u,-0.5,-0.2)] * TelecommutingSpace[0,1] +
b_ShuttleServ[(u,-0.6,-0.1)] * ShuttleService[0,1]




$

[Michiel Bliemer]

Your number of parameters is very large and therefore you need a much larger number of rows.
If you increase it to 24 or larger it is fine. Larger is always better because you get more variety in your data to estimate the parameters.

Note that you can replace your conditional constraints with a shortcut like X[X], see script below.

You can consider including a constant in increase or decrease. You can also consider using alternative-specific parameters for increase and decrease if you suspect that behaviour may be different.

Code: Select all

Design
;alts = decrease, maintain, increase
;rows = 24
;block = 4
;eff=(mnl,d)
;model:
U(decrease)= b_bikeshareSt[(u,0.2,0.5)] * FreeBikeShareSt[0,1] +
b_bikeshareM[(u,0.001,0.03)] * FreeBikeShareM[0,60,120] +
b_bikePark[(u,-0.5,-0.2)] * FreeBikePark[0,1] +
b_bikeElecSt[(u,0.27,0.625)] * FreeEbikeSt[0,1] +
b_bikeElecVo[(u,-0.05,-0.02)] * FreeEbikeVouch[0,10,30] +
b_bikeRepSt[(u,-0.21,-0.09)] * BikeRepairSt[0,1] +
b_bikeRepTu[(u,-1.5,-0.5)] * FreeBikeTunning[0,1] +
b_AccChildcare[(u,-0.32,-0.08)]* BikableAceessChildCare[0,1] +
b_AccSchool[(u,-0.32,-0.08)] * BikableAceessSchool[0,1] +
b_AccGrocery[(u,-0.5,-0.3)] * BikableAceessGrocery[0,1] +
b_AccHospital[(u,-0.32,-0.08)] * BikableAceessHospital[0,1] +
b_AccRestaurant[(u,-0.5,-0.1)] * BikableAceessRestaurant[0,1] +
b_AccGym[(u,-0.42,-0.03)] * BikableAceessGym[0,1] +
b_AccCommunity[(u,-0.25,-0.03)]* BikableAceessCommunity[0,1] +
b_AccBank[(u,-0.5,-0.3)] * BikableAceessBank[0,1] +
b_AccPetCare[(u,-0.32,-0.08)] * BikableAceessPetCare[0,1] +
b_Telecommute[(u,0.2,0.5)] * TelecommutingSpace[0,1] +
b_ShuttleServ[(u,0.1,0.6)] * ShuttleService[0,1]
/
U(maintain)= A[(u,-0.5,0.5)]
/

U(increase)= b_bikeshareSt[(u,-0.5,-0.2)] * FreeBikeShareSt[FreeBikeShareSt] +
b_bikeshareM[(u,-0.001,-0.03)] * FreeBikeShareM[FreeBikeShareM] +
b_bikePark[(u,0.2,0.5)] * FreeBikePark[FreeBikePark] +
b_bikeElecSt[(u,-0.625,-0.27)] * FreeEbikeSt[FreeEbikeSt] +
b_bikeElecVo[(u,0.02,0.05)] * FreeEbikeVouch[FreeEbikeVouch] +
b_bikeRepSt[(u,0.09,0.21)] * BikeRepairSt[BikeRepairSt] +
b_bikeRepTu[(u,0.5,1.5)] * FreeBikeTunning[FreeBikeTunning] +
b_AccChildcare[(u,0.08,0.32)] * BikableAceessChildCare[BikableAceessChildCare] +
b_AccSchool[(u,0.08,0.32)] * BikableAceessSchool[BikableAceessSchool] +
b_AccGrocery[(u,0.3,0.5)] * BikableAceessGrocery[BikableAceessGrocery] +
b_AccHospital[(u,0.08,0.32)] * BikableAceessHospital[BikableAceessHospital] +
b_AccRestaurant[(u,0.1,0.5)] * BikableAceessRestaurant[BikableAceessRestaurant] +
b_AccGym[(u,0.03,0.42)] * BikableAceessGym[BikableAceessGym] +
b_AccCommunity[(u,0.03,0.25)] * BikableAceessCommunity[BikableAceessCommunity] +
b_AccBank[(u,0.3,0.5)] * BikableAceessBank[BikableAceessBank] +
b_AccPetCare[(u,0.08,0.32)] * BikableAceessPetCare[BikableAceessPetCare] +
b_Telecommute[(u,-0.5,-0.2)] * TelecommutingSpace[TelecommutingSpace] +
b_ShuttleServ[(u,-0.6,-0.1)] * ShuttleService[ShuttleService]
$

Michiel

[Reihaneh]

Thank you so much for your prompt response; I really appreciate it. It was really helpful.

[Reihaneh]

Dear Dr. Bliemer,

I have another question regarding the design of my bike, car, and transit pass ownership options. As I mentioned, I have decided to create three separate designs for each tool, and I plan to combine the scenarios to get integrated ones, including all three ownership tools. In this way, some parameters are the same across these options. Is there a way to ensure consistent results for these parameters in each of these individual designs?

Thank you so much in advance,
Reihaneh

[Michiel Bliemer]

I am not sure what you mean with " consistent results for these parameters in each of these individual designs", are you talking about model estimation? In model estimation, you will pool the data and you can use a generic coefficient if you believe that a parameter should be the same across scenarios.

If you are referring to a parameter prior, you could use the same prior to generate each design, or you can use a different prior, depending on whether you believe the parameter would be generic or scenario/alternative-specific.

Michiel