Hi, we are currently working on a project that includes incremental training of LMs. Hence, there are plans to introduce quick adaptation in IRSTLM, but not soon.
The question is indeed how often you need to adapt the LM. If you are working with large news LMs then it seems that adapting once a week is enough (you simply do not collect enough data in fewer days to significantly change the LM). If you want to continuously update the LM you can also consider using an external interpolation. You interpolate two distinct LMs, one fixed and one smaller that is continuously retrained (should be fast to do), using the interpolate-lm command (see manual). Greetings, Marcello On Apr 22, 2012, at 9:12 PM, Pratyush Banerjee wrote: > Hi, > > I have recently been trying to create incremental adapted language models > using IRSTLM. > > I have a in-domain data set on which the mixture adapted weights are computed > using the -lm=mix option and i have a larger out-domain dataset from which i > incrementally add data to create adapted LMs of different size. > > Currently, every time saveBIN is called, the entire lmtable is estimated and > saved which makes the process slow... > > Is there a functionality in IRSTLM to incrementally train/save adapted > Language models? > > Secondly, given a existing adapted language model in ARPA format (old), and > another small language model built on incremental data (new), > > would it be safe to update the smoothed probabilities (fstar) using the > following formula: > c_sum(wh) = c_old(wh) + c_new(wh) > f*_old(w|h)*(c_old(wh)/c_sum(wh)) + f*_new(w|h)*(c_new(wh)/c_sum(wh)) > > where the c_old and c_new counts are estimated from the ngram tables? > > > Thanks and Regards, > > Pratyush > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
