Error message with RP model

[jbhokie1]

Hi,

I am trying to run the following random parameters model:

Design
;alts = A, B, C
;rows = 48
;eff = (rp,d)
;rdraws =random(1000)
;alg = swap(stop=total(100000 iterations))
;model:
U(A) = b[1] + b2[u,-.068,.478]*numbturb[1,2,3,4,5,6] + b3[u,-.772,.109]*height[276,390] + b4[u,-0.261,.105]*viewhome[0,1] + b5[u,-0.12,.0714]*noise[0,30,45] + b6[u,-.261,.105]*viewact[0,1] + b7[u,.002,.00147]*disc[1,5,10,15,20]/
$

using prior parameter estimates for my mean and standard deviation values. I get the following error message: "Error: The 'model' property contains a uniform random prior that has a lower bound that it larger than the Bayesian bound. 'u,.002,.00147'". I'm not entirely sure what this error means. Could anyone guide me as far as correcting it? Thank you!

[johnr]

Hello

There are a few errors in your syntax that require attention.

Firstly, you have three alternatives, but only one utility function for the first alternative. You can have two no-choice alternatives, however, you end the first utility function with a ' / ' which tells the program to expect another utility function which it cannot find. You then end the syntax with a $ .

Solution: a) If you do indeed require two no choices and a single alternative, you will need to replace the ' / ' with a ' $ '. Note that you are still probably not going to find a design as the priors will make the relative utilities such that the first alternative will probably be very undesirable relative to the two no choices.
b) You need to add a utility function for the second (and perhaps third alternative). The last utility function specified should end in a ' $ '.

Secondly, the ordering of the bounds for the uniform prior distribution specified for b7 is incorrect. It should be b7[u, lower bound, upper bound]. 0.002 is greater than 0.00147 in magnitude hence the bounds need to be switched.

John

[jbhokie1]

Thank you for your help.

I have edited my syntax based on your feedback. I have added a second utility function because my intent is to have 3 alternatives, one being a no-choice alternative:

Design
;alts = A, B, C
;rows = 48
;eff = (rp,d)
;rdraws =random(1000)
;alg = swap(stop=total(100000 iterations))
;model:
U(A) = b[1] + b2[n,-.068,.478]*numbturb[1,2,3,4,5,6] + b3[n,-.772,.109]*height[276,390] + b4[n,-0.261,.105]*viewhome[0,1] + b5[n,-0.12,.0714]*noise[0,30,45] + b6[n,-.261,.105]*viewact[0,1] + b7[n,.002,.00147]*disc[1,5,10,15,20]/
U(B) = b2*numbturb + b3*height + b4*viewhome + b5*noise + b6*viewact + b7*disc
$

The priors in my syntax are the mean followed by the standard deviation [distribution, mean, 2SD]. I realize now that I should have been indicating a normal distribution, not a uniform distribution, so I changed this. When I enter this new syntax into Ngene I get the following message:

Warning: No valid design has been found after 10 minutes. If no designs have been evaluated (see the counter below), this may be because a single design is taking a very long time to evaluate. If invalid designs are reported below, there may be a problem with the specification of the design. A common problem is that the choice probabilities are too extreme (close to 1 and 0), perhaps because some or all of the prior values are too large. Also, it is generally a good idea to start with a simple design (MNL, non-Bayesian), then add complexity. If you press stop, a design will be reported, which may assist in diagnosing the problem.

Do you have any advice on how to address this warning message? Thank you!

[johnr]

Hi

The problem is with b3. If you look at the attribute levels you have chosen, and the priors:

Attribute min max range Midpoint level Parameter Average utility Minimum utility Maximum utility
numbturb 1 6 5 3.5 -0.068 -0.238 -0.068 -0.408
height 276 390 114 333 -0.772 -257.076 -213.072 -301.08
viewhome 0 1 1 0.5 -0.261 -0.1305 0 -0.261
noise 0 45 45 22.5 -0.12 -2.7 0 -5.4
viewact 0 1 1 0.5 -0.261 -0.1305 0 -0.261
disc 1 20 19 10.5 0.002 0.021 0.002 0.04


You can see from the above that the average marginal utility for the height attribute is -257.076, whilst the next largest average marginal utility is -2.7. The height attribute is dominating utility and no matter what other level is selected for the other attributes, the priors you have chosen suggest that basically height is the only attribute being selected on. You need to scale the priors so they reflect the magnitudes of the attribute levels.

so rather than use b3[n,-.772,.109]*height[276,390] , try b3[n,-0.00772,0.00109]*height[276,390] which will work. See syntax below.

Design
;alts = A, B, C
;rows = 48
;eff = (rp,d)
;rdraws =random(1000)
;alg = swap(stop=total(100000 iterations))
;model:
U(A) = b[1] + b2[n,-.068,.478]*numbturb[1,2,3,4,5,6] + b3[n,-0.00772,0.00109]*height[276,390] + b4[n,-0.261,.105]*viewhome[0,1] + b5[n,-0.12,.0714]*noise[0,30,45] + b6[n,-.261,.105]*viewact[0,1] + b7[n,.002,.00147]*disc[1,5,10,15,20]/
U(B) = b2*numbturb + b3*height + b4*viewhome + b5*noise + b6*viewact + b7*disc
$

[jbhokie1]

Thank you for all of your help so far.

I am trying to do something similar to the post "Effects coded parameters & rp logit" submitted by nashkato, except rather than effects coding I'd like to use dummy coding. This is my code:

Design
;alts = A, B, C
;rows = 48
;eff = (rppanel,d)
;rep = 500
;rdraws = random (1000)
;model:
U(A) = b1[1] + b2.dummy[n,-.408,.478|n,-.34,.478|n,-.272,.478|n,-.204,.478|n,-.136,.478]*numbturb[6,5,4,3,2,1] + b3.dummy[n,-0.00772,0.00109]*height[390,276] + b4.dummy[n,-.261,.105]*viewhome[1,0] + b5.dummy[n,-.36,.0714|n,-.24,.0714]*noise[45,30,0] + b6.dummy[n,-.261,.105]*viewact[1,0] + b7[n,.002,.00147]*disc[1,5,10,15,20]/
U(B) = b2*numbturb + b3*height + b4*viewhome + b5*noise + b6*viewact + b7*disc
$

I'm getting the same error as before ("Warning: No valid design has been found after 10 minutes. If no designs have been evaluated (see the counter below), this may be because a single design is taking a very long time to evaluate. If invalid designs are reported below, there may be a problem with the specification of the design. A common problem is that the choice probabilities are too extreme (close to 1 and 0), perhaps because some or all of the prior values are too large. Also, it is generally a good idea to start with a simple design (MNL, non-Bayesian), then add complexity. If you press stop, a design will be reported, which may assist in diagnosing the problem.")

Is it a scaling issue again? With b2? If so, do you mind helping me by telling me how I should scale the priors in order to fix the problem?

Thank you!

[Michiel Bliemer]

There are several issues with what you are trying to do here.

First of all, you are using random draws, I would highly suggest more efficient draws such as halton or gaussian.
Secondly, you have a huge number of random parameters, each having a very wide standard deviation. You will near likely more than 100,000 draws to get stable results.
Thirdly, you have 48 rows but only ask for 500 repetitions. This will also be very unstable, you likely need more than 10,000 repetitions.

Having 100,000 rdraws and 10,000 reps will be unfeasible unless you can wait for about 100 years for Ngene to come up with an efficient design.

Fourthly, 48 rows is huge for a panel model, are you really giving a single respondent 48 rows? What happens internally is that for a panel model, the choice probabilities will be multiplied for each of the 48 choice tasks, which leads in the end to an extremely small probability, so small that the computer can no longer produce numerically accurate results.

Fifthly, I do not recommend trying to find an optimal design for an RP panel model for a model of such dimensions. A better approach would be to optimise for the MNL model using ;eff = (mnl,d), you can leave the rest of the syntax the same, and then select an MNL efficient design and inspect the properties for the RP panel model. We have found (see our paper Bliemer and Rose (2011) in Transportation Research Part B) that an MNL efficient design is usually also rather efficient for estimating RP panel models. However, in your case you first have to think about the number of rows, the number of random parameters and whether the priors for the standard deviation indeed are so large, and set proper values for reps and rdraws. I doubt you will be able to estimate this model if you try to include all parameters as random pararmeters, you may have to make a selection and keep some fixed parameters.

Michiel

[jbhokie1]

Michiel,

Thank you for your feedback. A single respondent will be given 4 choice sets, so I will change ;rows to equal 4. My advisor told me to use 2 standard deviations, but I could try reducing this to one SD. Given these changes, is there a number of rdraws and reps you recommend that I use?

Here is the revised syntax:

Design
;alts = A, B, C
;rows = 4
;eff = (rppanel,d)
;rep =
;rdraws =
;model:
U(A) = b1[1] + b2.dummy[n,-.408,.239|n,-.34,.239|n,-.272,.239|n,-.204,.239|n,-.136,.239]*numbturb[6,5,4,3,2,1] + b3.dummy[n,-0.00772,0.000545]*height[390,276] + b4.dummy[n,-.261,.0525]*viewhome[1,0] + b5.dummy[n,-.36,.0357|n,-.24,.0357]*noise[45,30,0] + b6.dummy[n,-.261,.0525]*viewact[1,0] + b7[n,.002,.000736]*disc[1,5,10,15,20]/
U(B) = b2*numbturb + b3*height + b4*viewhome + b5*noise + b6*viewact + b7*disc
$

I am also putting together an MNL model to try.

Thank you again for your help.

[jbhokie1]

I forgot to mention that we are sending our survey to 1,000 respondents.

[Michiel Bliemer]

Setting ;rows = 4 will not work, as you are trying to estimate 21 parameters, so you will need at least 21 data information points, and hence 21 rows. You can set ;rows = 24 and ;block = 6, such that each respondent will see 4 choice tasks.

I would suggest you read a bit more into the literature on experimental design, there are many issues here and you cannot randomly set these properties, designing an experiment takes time and effort and is not trial and error. The Ngene manual contains some theory that may give you a good start, as it will discuss the number of rows needed, the number of draws, etc.

You still have 10 random parameters which is huge, so you will need around 100,000 rdraws. For 24 rows, you will still need at least 10,000 reps I think. This is still very much infeasible, as you will be doing 1 billion computations PER DESIGN, and you will need to inspect thousands or millions of these designs to find an efficient one. This will take years of computation time.

I ran your syntax for estimating an MNL model, and it seems to suggest with your parameters (just the mean, not the standard deviation) that you will be able to estimate all parameters with 1000 respondents, except b3 and b7 which have a very small value and will be difficult to estimate statistically significant. Note that the sample size computations of Ngene assume that the priors are correct, so it could be that the priors of b3 and b7 are not close to the true value and that you will be able to estimate them statistically significant. Estimating an MNL model is one thing, for the RP panel model you need many more respondents, as you are estimating twice the number of parameters. One thousand respondents is in that case not very much. My suggestion again is to first create an MNL efficient design, you will not be able to find an RP panel efficient design with your dimensions.

Michiel