Ngene 1.1.2 memory leak

[paulm]

Andrew,

I just started using the new Ngene version 1.1.2. I ran a design for around 3 hours before I got the following:
TERMINAL ERROR: Out of Memory.
Aborted.

I was able to save a design by choosing 'continue' in a resulting dialog box. But that prevented me from saving other crash information.

Any suggestions?

Paul

[paulm]

Following up, Ngene doesn't seem to recognize ;eval=<path>, newblocking. As far as I can tell, it treats ', newblocking' as part of the file name.

And can ;eval= read a .ngd design file?

Ngene is really bad at blocking. What I would like to do is a two-step process. Step 1, create a design. Step 2, reblock the design with better blocking. But I"m not sure how to implement step 2.

[Michiel Bliemer]

Thanks Paul, we will look into it as soon as possible.

Regarding blocking, finding a blocking column is only really possible with orthogonal designs (generating them from scratch). For efficient designs, Ngene tries to find a blocking column which minimises correlations with the attribute levels. As far as I am aware, blocking of efficient designs has not been discussed in the literature, the technique in Ngene may not be the method that is best for everyone as it may depend on the attribute levels, model, constraints, etc. We are working on other algorithms that aim at optimising other objectives for finding blocking columns. Blocking columns that I use are mostly very good, but It very much depends on your design and what you think to optimise in blocking (as a true blocking column often does not exist).

Based on requests, in version 1.1.2 we have added the option to keep the blocking column when reading in a design. The command 'newblocking' was the default in the previous Ngene version. You should be able to read in an .ngd design. I will ask Andrew to respond to your error message.

[Andrew Collins]

Hi Paul

In terms of the memory issue, can you send me your syntax please? If you don't want to post it here, you can send it to contact@choice-metrics.com, and I can keep the specifics of the design specification out of this forum when I reply.

Yes, you can generate a new blocking column. I have inadvertently put the newblocking option under the wrong property in the manual. It should be an argument of the block property, e.g. ;block=4,newblocking. Thanks for alerting me to this, I will update the manual.

The eval property can be used to evaluate an existing design, either with a full path, or just the design name if the design is saved to the current project.

By default, Ngene will only spend 2 seconds generating the blocking column. This number was chosen as the default because the blocking is only performed when the design is opened, and any longer would make opening a design quite unresponsive. You put arguments in the block property to increase how much time is spent on blocking (e.g. ;block=4,minmax,noimprov(5 mins) ), where this could be used either during design generation, or later in a two step process as you suggest. We have fixed an earlier bug that prevented the time from being more than 58 seconds. In terms of the blocking performance, I have generally found that the maximum correlations between the blocking column and each of the attributes has been low, less than about 0.05. But I have found that if each block has only a small number of choice tasks, then attribute level balance may be poor for at least some of the choice tasks.

Andrew

[paulm]

Thanks Andrew,

I have sent in the script generating the memory error.

And moving ,newblocking seems to work as expected.

To all other researchers on designs: it certainly appears that blocking is not a topic that is researched much. But it is an important topic and further research is needed. This is particularly true when using random effects models.

The method used by Ngene for blocking may well be best. Perhaps we are just not letting it run for long enough. In particular, I believe (although I do not have citations to prove it) that it is important for each block to contain each level of each attribute. Preferably more than once. Otherwise, you are essentially using data augmentation for missing data in you RP model. With insufficient time devoted to blocking, this condition is not assured.

I welcome further suggestions although perhaps this dicussion belongs in the other forum.

Paul