Hi - We switched to using IRSTLM recently, in order to build bigger language models. I am starting to think, however, that the entire model is still being loaded into memory. Here's part of what Moses prints out now:
Start loading LanguageModel /net/tidesserver/tidesserver_raid7/clasr/ john/mt4st/experiments/mt4st.NEWLM1/working-dir/lm/mitrebnbc.blm : [33.000] seconds In LanguageModelIRST::Load: nGramOrder = 5 Loading LM file (no MAP) Qblmt loadbin() reading 256 centers reading 256 centers reading 256 centers reading 256 centers reading 256 centers loading 483595 1-grams loading 6753166 2-grams loading 29675357 3-grams loading 57676563 4-grams loading 70527466 5-grams done OOV code is 483594 IRST: m_unknownId=483594 Set dictionary_upperbound to: 483596 Set logOOVpenalty to: 0.000 Finished loading LanguageModels : [55.000] seconds Does "no MAP" indeed mean that the whole LM is being loaded? The code at that point (trunk/moses/src/LanguageModelIRST.cpp) seems to distinguish between getting called with one vs two filenames, where the second one is a "micro-macro map file". We are just passing the binary .blm file into Moses - is this incorrect? Thanks for any advice. - John Burger MITRE _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
