reality based dominance in efficient design

[alexsydney]

Hi
I am Alex Karki from Sydney.
Thank you for Ngene manual which is easy to understand.

I have two alternatives A & B and following attributes to choose parking.
1. Walk time to destination
2. Parking cost
3. Finding parking space(search time)

Relation between Attributes.
1. If Parking Cost increase then Walk time & finding parking space time to be less
2. If Parking Cost decrease then Walk time & finding parking space time to be more.

I undertook efficient design and constrain efficient design but still I get majority of scenarios to have one alternative as OBVIOUS dominant.

Reading through Ngene manual, efficient design picks two extreme attribute levels to minimize VC matrix.

Constrain Efficient Design code:
Design
;alts = A, B
;rows = 24
;eff = (mnl,d)
;cond:
if(A.cost=[20,25], A.walk=2),
if(A.cost=0, A.walk=11),
if(A.cost=[10,5], A.walk=8),
if(A.cost=15, A.walk=5),
if(A.cost > B.cost, A.find <= B.find) ,
if(B.cost=[20,25], B.walk=2),
if(B.cost=0, B.walk=11),
if(B.cost=[10,5], B.walk=8),
if(B.cost=15, B.walk=5)
;model:
U(A) = b2[-0.001] * walk[2,5,8,11] + b3[-0.002] * cost[0, 5, 10,15,20,25] + b4[-0.003] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find $

without constrain design (Efficient Design)
Constrain Efficient Design code:
Design
;alts = A, B
;rows = 24
;eff = (mnl,d)
;model:
U(A) = b2[-0.001] * walk[2,5,8,11] + b3[-0.002] * cost[0, 5, 10,15,20,25] + b4[-0.003] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find $

Question
1. Can I hand-pick and include only the non-dominant scenarios among the 24 rows ?

[Michiel Bliemer]

Dominant alternatives can be easily removed automatically in Ngene by using the * behind generic alternatives, see below. I admit that this is a bit hidden in the manual (page 184). The syntax below will generate a design without strict dominancy (which only depends on the sign of the priors. Weak dominancy (predicted by the probabilities) depends on the exact values of the priors, so if you also want weakly dominant alternatives to be removed, you should set the priors to values that are accurate (i.e. come from a pilot study). If you do, I would recommend using Bayesian priors.

Code: Select all

Design
;alts = A*, B*
;rows = 12
;eff = (mnl,d)
;model:
U(A) = b2[-0.001] * walk[2,5,8,11] + b3[-0.002] * cost[0, 5, 10,15,20,25] + b4[-0.003] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find
$


Note that finding 24 rows without a dominant alternative is quite difficult with the swapping algorithm. You can use the modified federov algorithm instead, but this does not guarantee attribute level balance. You can impose restrictions on level balance (e.g., find[1,3,5](6-10,6-10,6-10))

Code: Select all

Design
;alts = A*, B*
;rows = 24
;eff = (mnl,d)
;alg = mfederov
;model:
U(A) = b2[-0.001] * walk[2,5,8,11] + b3[-0.002] * cost[0, 5, 10,15,20,25] + b4[-0.003] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find
$


[alexsydney]

thank you for your reply.
The syntax below gives 12 scenarios, only 6 scenario are neutral but remaining 6 are dominant.

*** Can I only pick 6 scenarios from the 12 scenarios ? If I do this, Can this still be efficient design or simply fractional factorial ?

Design
;alts = A*, B*
;rows = 12
;eff = (mnl,d)
;alg=mfederov
;model:
U(A) = b2[-0.001] * walk[2,5,8,11] + b3[-0.002] * cost[0, 5, 10,15,20,25] + b4[-0.003] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find
$

[Michiel Bliemer]

If you put in the correct sign of the priors in the syntax, Ngene WILL not generate strictly dominant alternatives. A strictly dominant alternative is better in each attribute.

If you put in the correct value of the priors in the syntax, Ngene WILL not generate weakly dominant alternatives. A weakly dominant alternative has a high choice probabilitie (e.g 90%).

Note that alternatives with the same choice probabilities (e.g., 50-50%) are NOT efficient to ask. More efficient are choice tasks that have one alternative somewhat more attractive than the other (e.g., 70-30% or 60-40%). This is not just a belief, but this has been mathematically and statistically proven (work by Kanninen, Johnson, etc.). So if you mean with neutral 'utility balanced', then this may not be a good idea to look at. Let Ngene do the work for you and optimise the design. You merely have to provide good prior parameters as input.

If you see any dominant alternatives, then your priors are incorrect. Note that when you manually check for dominancy, you have some priors in your mind that you use for judgement. It looks like -0.001 and -0.002 are far from the true values.

So invest some time in getting better priors that make more sense, simply by your own judgement, and use Bayesian priors to make the design more robust. I would not suggest any manual procedures to simply remove choice tasks that you do not like. You can use willingness-to-pay from the literature to convert to priors for example.

[alexsydney]

Thank you for your reply.

This is from your previous post
"2. Each design will be different and there is no value of D-error that is good or bad, this is case specific. It usually helps best to look at the S-estimates, which tell you the likely sample size needed to estimate the parameters (assuming that your priors are correct). If these S-estimates are reasonable, say lower than 50 or 100, then it should be fine. If they produce values over 1000, then you may wish to reconsider your design or doubt your priors."

My s-estimates are above 120,000, this means I need sample size of 120,000 ? Which is basically impossible.

Bayesian priors, what range does it have like is it -1000 to +1000 or only within decimal points.

If these priors are the coefficients calculated in program like stata then priors varies within the likelihood of -2 to +2 for most cases?

[Michiel Bliemer]

Your S-estimates only make sense if your priors are sufficiently accurate. So I repeat my previous post, you need to provide better priors. I suggest you do a pilot study. A pilot study will provide you with Bayesian priors, which are normally distributed with the mean equal to your estimated coefficient from estimation software and the standard deviation set to your standard error. I am not sure what you mean "within the likelihood of -2 and +2".

[alexsydney]

thank you for your reply.

I conducted small pilot study with sample size n=10

"Find" attribute got omitted in the estimation software so I had to guess estimate the bayesian prior for 'Find' attribute.

**Is there any reasons why this attribute could be omitted ?

*** Should I include the intercept or b1 in syntax below ? current syntax has intercept/b1 = 0

Design
;alts = A*, B*
;rows =12
;eff = (mnl,d)
;block = 2
;model:
U(A) = b2[n,-0.6486367,0.3414198] * walk[2,6,10] + b3[n,-0.3522991,0.2165178] * cost[0, 5, 10,15,20,25] + b4[n,-0.1,0.03] * find[1, 3, 5] /
U(B) = b2 * walk + b3 * cost + b4 * find $

[Michiel Bliemer]

You are now using random parameters in your syntax, assuming a mixed logit model. You should be used Bayesian priors in a Bayesian design context. So please use b2[(n,-0.6486367,0.3414198)] with brackets around the prior values.

To answer your non-Ngene questions, I do not understand what you mean by the 'find' attribute got omitted from the software? I am not able to answer specific questions on other estimation software. Perhaps you mean not statistically significant? In that case you can still use estimates as priors.
On your other question, your experiment seems unlabelled, so you should not have an intercept unless you are interested in left-to-right or top-to-bottom bias.

[alexsydney]

thank you for your post, it has been helpful in my understanding.

I have few basic queries, please advise if I should use another forum.

b3[(n,-0.3522991,0.2165178)] * cost[0, 5, 10,15,20,25]

-0.352 (This value is coefficient from estimation software, This coefficient is log of odds ?

After (0.352/0.216=1.62 ) 1.62 standard deviation, this means 94.74% cost will have negative sign and remaining 5.26% prefer to still choose the high price. This might be from other attribute like high walk time or finding parking space time.

I have already conducted two pilot survey, so getting third pilot survey is difficult.

[Michiel Bliemer]

What do you mean, "this coefficient is log of odds?" I do not understand what that means.

I don't know what 0.216 means, and what do you mean with standard deviation, do you mean standard error? Or perhaps you are estimating a mixed logit model?

Sorry these are not experimental design questions, so I am afraid will have to direct you the manual of the estimation software you are using.

To summarise, you will need to use the estimated coefficient of the multinomial logit model as the prior mean and the standard error as the prior standard deviation assuming it is normally distributed, so b3[(n,coefficient,standard error)].

A normal distribution will always have a tail on both sides, so positive and negative. If you want to avoid that, you can use a uniform distribution, so b3[(u,-0.5,0)], which means it is always negative.