choice-metrics.com

Hi,

my questions relate to iteration time for d-efficient designs with priors and if there are potential strategies to reduce the time to access first design solutions.

I am a beginner with Ngene and found it very easy to use for developping different orthogonal designs, which always had very short iteration times (e.g. several minutes).
Based on pre-test estimates of an orthogonal design I wanted to develop a d-effecient design with 6 attributes between 2 to 4 levels (see code at the end of the post).

Ngene has now been running for 48 hours, completing close to 4 million evaluations, but so far it has not produced any preliminary result in the iteration history field, which I could access.
- I observe that iterations for row swapping with a new seed have steadily increased from initially less than 20,000 swaps to now 55,000 swaps
- I cannot see that the seeds used to converge (last 8 seeds: 0.075447, 0.076756, 0.073022, 0.075099, 0.075603, 0.074021, 0.073013, 0.078329)

My questions are:
a) from your experience is there a benchmark for the time requird for a first iteration result of a d-efficient design? E.g. is it normal to take more than 2 days?
b) does the developement of seeds and row swaps described above indicate how much more time is required until a first design solution is identified?
c) are the ways to access preliminary solutions, which are not yet shown in the iteration history field?
d) are the specific session options, which allow a less accurate but faster design generations?

I could not find related information in the manual, maybe I just did not see it.

Thank you in advance for your advice on this!
Simone

Code: Select all

Design
;alts = alt1, alt2, alt3, alt4
;rows = 48
;eff = (mnl,d)
;model:
U(alt1) = b.effects [0.15|0.08|-0.36] * B[0,1,2,3] +
          a.effects [0.19|0.43|-0.37] * A[0,1,2,3] +
          f.effects [-1.22|-0.21|0.49] * F[0,1,2,3] +
          o.effects [-1.13] * O[0,1] +
          k.effects [-0.82] * K[0,1] +
          p.effects [1.57|0.67|-0.77] * P[0,1,2,3] /

U(alt2) = b.effects [0.15|0.08|-0.36] * B[0,1,2,3] +
          a.effects [0.19|0.43|-0.37] * A[0,1,2,3] +
          f.effects [-1.22|-0.21|0.49] * F[0,1,2,3] +
          o.effects [-1.13] * O[0,1] +
          k.effects [-0.82] * K[0,1] +
          p.effects [1.57|0.67|-0.77] * P[0,1,2,3] /

U(alt3) = b.effects [0.15|0.08|-0.36] * B[0,1,2,3] +
          a.effects [0.19|0.43|-0.37] * A[0,1,2,3] +
          f.effects [-1.22|-0.21|0.49] * F[0,1,2,3] +
          o.effects [-1.13] * O[0,1] +
          k.effects [-0.82] * K[0,1] +
          p.effects [1.57|0.67|-0.77] * P[0,1,2,3] /

U(alt4) = b.effects [0.15|0.08|-0.36] * B[0,1,2,3] +
          a.effects [0.19|0.43|-0.37] * A[0,1,2,3] +
          f.effects [-1.22|-0.21|0.49] * F[0,1,2,3] +
          o.effects [-1.13] * O[0,1] +
          k.effects [-0.82] * K[0,1] +
          p.effects [1.57|0.67|-0.77] * P[0,1,2,3] $


Hi Simone

Thanks for the feedback.

I will answer your questions in turn.

1. Typically what I do is let it run over night. There are usually billions of possible designs, which would take years to test them all. We have found that you get huge gains in efficiency searching say the first 10,000 or so after which things tend to slow down. Note that this is not a hard or fast number and may vary depending on circumstance. Depending on the design type, it may take shorter or longer - for example, if you have lots of Bayesian priors, it may take a few days just to check a few 100 designs - I wouldn't let it run more than a few days in this case unless you have a lot of time - you might be far from optimal, but it should be far from the worst design after this time. We are working on a new range of algorithms, and comparing how these perform at present - we plan on releasing these in a later release version at some stage. We have compared these to the existing algorithms in Ngene and the results can be found in a working paper ITLS-WP-11-19 located here http://sydney.edu.au/business/itls/rese ... ing_papers. This paper will hopefully give you an insight into relative performance of the algorithms in locating a design.

2. No, each algorithm, it starts with a random design and applies the logic of that algorithm. For example, if you apply the RSC algorithm, it will start with a random design and then work through, R, then S then C. Once it has searched through all possible combinations, or if it has not found a better design after so many checks, it will reseed meaning it selects a new random design and starts again. This has nothing to do with time to find a design, as the original random design might be a good design already, or it could be a really bad one.

3. I'm not sure what you mean here. Once Ngene has found a design, it stores it. However, storing lots of designs is memory intensive and may cause problems. Hence, we have defaulted to storing only the first design found and then the best 9 or 10 designs thereafter. You can store the best X number of designs and look at these by adding the command store=50 where X = 50 in this case. At any time, you can open up and look at these designs, however if it is a Bayesian design, or a complex model design, it may take a while to do this, as it will have to redo all the draws again (it just saves the X, so it will need to recalculate the efficiency measures, etc.).

4. No, however in a design with Bayesian priors, using less draws will do this (that is, be less accurate). The trick is that we are not going to search all possible designs unless you have years to do so. Hence, we search only a subset and hopefully find a very good one in that subset). When I ran your design on my PC, it was searching about 30 a second which to me would be pretty good speed anyway.

John