NGene design in Sawtooth Software

[Andrew]

Hi...

I am quite new to NGene and have a question concerning using a NGene-created design in Sawtooth Software.
I created an efficient mnl-design with NGene, 6 Attr/3 Level, 1 Attr/ 2 Level, 2 blocks, no prohibitions, no priors. Following formatting rules of Sawtooth I imported it in Sawtooth SSI Web.

This from NGene:

alt1.a alt1.b alt1.c alt1.d alt1.e alt1.f alt1.g alt2.a alt2.b alt2.c alt2.d alt2.e alt2.f alt2.g Block
1 1 0 2 2 1 0 1 1 2 0 0 1 1 2
1 1 2 0 2 2 1 1 1 0 2 0 0 0 1
....

Became this in Sawtooth after re-sorting blocks order:

Block,task,alt,Att_a,Att_b,Att_c,Att_d,Att_e,Att_f,Att_g
1,1,1,2,2,3,1,3,3,2
1,1,2,2,2,1,3,1,1,1
1,2,1....
1,2,2....
2,1,1,2,2,1,3,3,2,1
2,1,2,2,2,3,1,1,2,2
2,2,1....
2,2,2....
....

Everything seemed to be fine so far. The imported design was accepted. But later logit computation came up with some issues. Design was spotted as deficient. I didnt get any std.error. And another curiosity is the fact that coefficient of every 1st level is always exactly double than effect of the 2nd level, e.g. 1st level 0,462, 2nd level 0,231, 3rd, -0,693. Same ratio over all attributes. That cant be coincidence. Why didnt Sawtooth recognize the design as efficient?!

Any idea what went wrong here?!

I appreciate any help on this.

Best
Andy

[johnr]

Hi Andy

Unfortunately we are not Sawtooth users so it is difficult for us to state exactly what the software is doing. However, having attended many of their conferences in the past, it appears that they mainly assume zero priors when generating their designs. This leads to orthogonal or near orthogonal designs, assuming unlabeled experiments. I recall siting through one presentation at a Sawtooth conference where the presenter (who shall remain nameless) compared the empirically obtained D-errors resulting from data collected using Sawtooth designs to the empirically obtained D-errors of orthogonal designs across several comparison data sets and concluding that there was no difference between the two. Of course, the irony was that the presenter was using zero priors in unlabelled experiments to construct their D-efficient designs, which will result in orthogonal (or near) designs and hence was concluding that orthogonal or near orthogonal designs will produce the same or similar results in terms of efficiency to orthogonal designs. Everyone sat their at the conference nodding away at this astounding revelation. Anyway I digress.

We would need to see the design and syntax used before we can comment on the efficiency of the design, and the data before we could comment on the outcomes.

John

[Michiel Bliemer]

It sounds like the assumptions you made in Ngene may be inconsistent with the assumptions you (implicitly?) made in Sawtooth. I do not have experience with Sawtooth, but from what you write it seems that there are some underlying assumptions in Sawtooth that are not consistent with what you have explicitly assumed in Ngene.

A few things to consider:
1. Is your experiment labelled or unlabelled?
2. What coding structure do you use? If you specified linear effects in Ngene, with only a single parameter per attribute, then this would be different from assuming dummy or effects coding that would estimate multiple parameters. You mention different coefficients per level, so that means you are trying to use dummy or effects coding (or another nonlinear coding type) in Sawtooth? Did you specify the same coding in Ngene? If your attribute has 3 levels, you will estimate 2 dummy or effects coded variables, or 3 orthogonal/orthonormal coded variables. So does Sawtooth use orthogonal coding, as you say that it states 3 coefficients for 3 levels?
3. If you use zero priors in Ngene, how does Sawtooth get the parameter estimates? Based on what data? There is no data (choice observations), so the coefficients cannot be estimated yet?
4. How many rows have you used in the design? You have 7 generic parameters to estimate (or 14 alternative specific parameters, depending on whether you are doing a labelled or unlabelled experiment) if you have a linear effects model. For dummy coding, this would mean 14 (or 28) parameters. If your design is generated under a linear effects model, but Sawtooth thinks you want to estimate 14 parameters, then you may need more rows in your design.

If you know exactly what model Sawtooth tries to estimate under which assumptions, you can generate an efficient design that will also be efficient according to Sawtooth. But you will have to make the model and the assumptions explicit, otherwise I am afraid I cannot comment, as I do not know Sawtooth and is therefore a black box to me.

Michiel

[Andrew]

Thanks John and Michiel for the very quick reply. I appreciate it.

I gonna test the avenues you mentioned. And I'd like to get back to you later.

Many thanks.

Andrew

[dsr]

Andrew

Can you post your ngene syntax?

One can import any design into Sawtooth SSIWEB - in that sense it's just a survey software well equipped to host choice tasks - it has a design element but you've opted to not to use it.

I think the things to check are:
1-your syntax and the design properties of your design - which can be checked here
2-your importing into SSIWEB of your ngene design (that's one for you)
3-the problematic model results- I think these are just using Sawtooth's default: simple (aggregate) Logit model, which means that the issue is unlikely to be that, but you can easily export your data into a stats package and re-estimate to check that

I'd start with 1&2, and if 1 looks ok then I suspect 2 is the issue (which is my guess at this stage)

Dan