Re: [Moses-support] moses may 10
we don't have loads of old experiments we need to rerun but if you do, make the change. It sounds reasonable. On 12 May 2010 13:58, Christian Hardmeier wrote: > Hi Hieu > > > chris - we can add a default=1 but talked to pwilliams about it, would > > need to keep the back compatiblity in the mert scripts & other places > > too. Would just be a source of constant bugs. > > I just discussed the matter with my office mate Arianna. We would favour > the following solution: > - default of 1 for the decoder > - error message about new format from MERT and other scripts > > The reason is that we have lots of fully trained old systems lying > around that we'd like to use without modifications. We also have old > patched > versions of the decoder, which we sometimes still use. > On the other hand, we don't think MERT needs to be able to deal with the > old format. If I retrain a system, I can also adapt the config file > format. > > Does this sound reasonable to you? > > /Christian > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Hi Hieu > chris - we can add a default=1 but talked to pwilliams about it, would > need to keep the back compatiblity in the mert scripts & other places > too. Would just be a source of constant bugs. I just discussed the matter with my office mate Arianna. We would favour the following solution: - default of 1 for the decoder - error message about new format from MERT and other scripts The reason is that we have lots of fully trained old systems lying around that we'd like to use without modifications. We also have old patched versions of the decoder, which we sometimes still use. On the other hand, we don't think MERT needs to be able to deal with the old format. If I retrain a system, I can also adapt the config file format. Does this sound reasonable to you? /Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
On Tuesday 11 May 2010 22:54:20 Hieu Hoang wrote: > there is a error message from the decoder. > ERROR:Expected at least 5 tokens per entries in 'ttable-file', but > only found 4 Hi Hieu Yes, when I first saw that error message I was puzzled by it. I would suggest something that explicitly mentions the change in the ini file format, and which gives an example of the new format, cheers Barry ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
On 5/11/2010 17:54, Barry Haddow wrote: > Hi Giuseppe > > The first error that you give is because the format of the moses.ini file has > changed. You need to add an extra digit at the beginning of the line that > specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a > binarised ttable. > > This worked, thank you. I would suggest that the comment in the moses.ini file at least would mention the additional field. > The second error is from irstlm. Probably it's because irstlm is not thread- > safe, and its cache has been corrupted by using it with mosesmt. You should > switch to srilm if you want to use mosesmt, > Using moses instead of mosesmt worked. Thank you -- Beppe > regards > Barry > > On Tuesday 11 May 2010 16:37:56 Giuseppe Attardi wrote: > >> We are running into problems with the version of moses released on April >> 26th. >> >> mosesmt fails when using tables prouced with a previous version: >> ... ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
there is a error message from the decoder. ERROR:Expected at least 5 tokens per entries in 'ttable-file', but only found 4 however, it's lost in all the credits and param explanation. So I've moved those out of the way when there's an error so that we can see message properly chris - we can add a default=1 but talked to pwilliams about it, would need to keep the back compatiblity in the mert scripts & other places too. Would just be a source of constant bugs. On 11/05/2010 17:28, Barry Haddow wrote: > Maybe a more transparent error message would help? > > On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: > >> i thought about making it back-compatible but the code gets messy and >> error prone. Theres now 3 more phrase table - the text SCFG, binary >> SCFG, and the suffix array. >> >> So i thought it better to take the punch now and feel a short, sharp >> pain rather than let it linger. >> >> however, anyone wants to put back the old code to make it back comp, >> they're welcome to, as long as u look after it >> >> On 11/05/2010 17:04, Christian Hardmeier wrote: >> >>> Hi, >>> >>> The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. >>> Is there a reason why we can't have backwards compatibility here? I'm a >>> bit concerned about moving to the latest decoder version since it will >>> require me to update the configuration file of each and every system >>> I've ever trained, and then they won't work with the old decoders any >>> more. Couldn't the decoder figure out on its own whether it should be 0 >>> or 1 if the indication is missing, as it used to do? >>> >>> Cheers, >>> Christian >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
On 11 May 2010 17:33, Christian Hardmeier wrote: > For my purposes, even a hard-coded assumption of 1, along with a more > transparent error message if the model isn't found, would do. Does > anybody actually decode with in-memory phrase tables in real life? > (well, I suppose some people do...) Google and anyone who actually wants to do more than optimise against a fixed dev/test set You can't afford to filter the phrase table when dealing with any old translation request Miles > > /Christian > > On Tue, 11 May 2010, Barry Haddow wrote: > >> Maybe a more transparent error message would help? >> >> On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: >> > i thought about making it back-compatible but the code gets messy and >> > error prone. Theres now 3 more phrase table - the text SCFG, binary >> > SCFG, and the suffix array. >> > >> > So i thought it better to take the punch now and feel a short, sharp >> > pain rather than let it linger. >> > >> > however, anyone wants to put back the old code to make it back comp, >> > they're welcome to, as long as u look after it >> > >> > On 11/05/2010 17:04, Christian Hardmeier wrote: >> > > Hi, >> > > >> > >> The first error that you give is because the format of the moses.ini >> > >> file has changed. You need to add an extra digit at the beginning of the >> > >> line that specifies the ttable-file. Add 0 for a memory-based ttable, >> > >> and 1 for a binarised ttable. >> > > >> > > Is there a reason why we can't have backwards compatibility here? I'm a >> > > bit concerned about moving to the latest decoder version since it will >> > > require me to update the configuration file of each and every system >> > > I've ever trained, and then they won't work with the old decoders any >> > > more. Couldn't the decoder figure out on its own whether it should be 0 >> > > or 1 if the indication is missing, as it used to do? >> > > >> > > Cheers, >> > > Christian >> > > ___ >> > > Moses-support mailing list >> > > Moses-support@mit.edu >> > > http://mailman.mit.edu/mailman/listinfo/moses-support >> > >> > ___ >> > Moses-support mailing list >> > Moses-support@mit.edu >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> > >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Maybe adding another option in moses would be an good idea, at least you don't need to modify the ini files. Just modify the commands would be easier i guess. Best regards! Jie Jiang CNGL, School of Computing, Dublin City University, Glasnevin, Dublin 9. Tel: +353 (0)1 700 6724 2010/5/11 Christian Hardmeier > For my purposes, even a hard-coded assumption of 1, along with a more > transparent error message if the model isn't found, would do. Does > anybody actually decode with in-memory phrase tables in real life? > (well, I suppose some people do...) > > /Christian > > On Tue, 11 May 2010, Barry Haddow wrote: > > > Maybe a more transparent error message would help? > > > > On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: > > > i thought about making it back-compatible but the code gets messy and > > > error prone. Theres now 3 more phrase table - the text SCFG, binary > > > SCFG, and the suffix array. > > > > > > So i thought it better to take the punch now and feel a short, sharp > > > pain rather than let it linger. > > > > > > however, anyone wants to put back the old code to make it back comp, > > > they're welcome to, as long as u look after it > > > > > > On 11/05/2010 17:04, Christian Hardmeier wrote: > > > > Hi, > > > > > > > >> The first error that you give is because the format of the moses.ini > > > >> file has changed. You need to add an extra digit at the beginning of > the > > > >> line that specifies the ttable-file. Add 0 for a memory-based > ttable, > > > >> and 1 for a binarised ttable. > > > > > > > > Is there a reason why we can't have backwards compatibility here? I'm > a > > > > bit concerned about moving to the latest decoder version since it > will > > > > require me to update the configuration file of each and every system > > > > I've ever trained, and then they won't work with the old decoders any > > > > more. Couldn't the decoder figure out on its own whether it should be > 0 > > > > or 1 if the indication is missing, as it used to do? > > > > > > > > Cheers, > > > > Christian > > > > ___ > > > > Moses-support mailing list > > > > Moses-support@mit.edu > > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > ___ > > > Moses-support mailing list > > > Moses-support@mit.edu > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > ___ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
For my purposes, even a hard-coded assumption of 1, along with a more transparent error message if the model isn't found, would do. Does anybody actually decode with in-memory phrase tables in real life? (well, I suppose some people do...) /Christian On Tue, 11 May 2010, Barry Haddow wrote: > Maybe a more transparent error message would help? > > On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: > > i thought about making it back-compatible but the code gets messy and > > error prone. Theres now 3 more phrase table - the text SCFG, binary > > SCFG, and the suffix array. > > > > So i thought it better to take the punch now and feel a short, sharp > > pain rather than let it linger. > > > > however, anyone wants to put back the old code to make it back comp, > > they're welcome to, as long as u look after it > > > > On 11/05/2010 17:04, Christian Hardmeier wrote: > > > Hi, > > > > > >> The first error that you give is because the format of the moses.ini > > >> file has changed. You need to add an extra digit at the beginning of the > > >> line that specifies the ttable-file. Add 0 for a memory-based ttable, > > >> and 1 for a binarised ttable. > > > > > > Is there a reason why we can't have backwards compatibility here? I'm a > > > bit concerned about moving to the latest decoder version since it will > > > require me to update the configuration file of each and every system > > > I've ever trained, and then they won't work with the old decoders any > > > more. Couldn't the decoder figure out on its own whether it should be 0 > > > or 1 if the indication is missing, as it used to do? > > > > > > Cheers, > > > Christian > > > ___ > > > Moses-support mailing list > > > Moses-support@mit.edu > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > ___ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: > i thought about making it back-compatible but the code gets messy and > error prone. Theres now 3 more phrase table - the text SCFG, binary > SCFG, and the suffix array. > > So i thought it better to take the punch now and feel a short, sharp > pain rather than let it linger. > > however, anyone wants to put back the old code to make it back comp, > they're welcome to, as long as u look after it > > On 11/05/2010 17:04, Christian Hardmeier wrote: > > Hi, > > > >> The first error that you give is because the format of the moses.ini > >> file has changed. You need to add an extra digit at the beginning of the > >> line that specifies the ttable-file. Add 0 for a memory-based ttable, > >> and 1 for a binarised ttable. > > > > Is there a reason why we can't have backwards compatibility here? I'm a > > bit concerned about moving to the latest decoder version since it will > > require me to update the configuration file of each and every system > > I've ever trained, and then they won't work with the old decoders any > > more. Couldn't the decoder figure out on its own whether it should be 0 > > or 1 if the indication is missing, as it used to do? > > > > Cheers, > > Christian > > ___ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: > Hi, > > >> The first error that you give is because the format of the moses.ini file has >> changed. You need to add an extra digit at the beginning of the line that >> specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a >> binarised ttable. >> > Is there a reason why we can't have backwards compatibility here? I'm a > bit concerned about moving to the latest decoder version since it will > require me to update the configuration file of each and every system > I've ever trained, and then they won't work with the old decoders any > more. Couldn't the decoder figure out on its own whether it should be 0 > or 1 if the indication is missing, as it used to do? > > Cheers, > Christian > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Hi, > The first error that you give is because the format of the moses.ini file has > changed. You need to add an extra digit at the beginning of the line that > specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a > binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Hi Giuseppe The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. The second error is from irstlm. Probably it's because irstlm is not thread- safe, and its cache has been corrupted by using it with mosesmt. You should switch to srilm if you want to use mosesmt, regards Barry On Tuesday 11 May 2010 16:37:56 Giuseppe Attardi wrote: > We are running into problems with the version of moses released on April > 26th. > > mosesmt fails when using tables prouced with a previous version: > > Defined parameters (per moses.ini or switch): > config: moses.ini > distortion-file: 0-0 msd-bidirectional-fe 6 > combinedBaseline/workdir/model/reorderTable/reordering-table > distortion-limit: 6 > input-factors: 0 > lmodel-file: 1 0 5 baseline/languageModel/english.blm.mm > mapping: 0 T 0 > ttable-file: 0 0 5 > combinedBaseline/workdir/model/phraseTable/phrase-table > ttable-limit: 20 > weight-d: 0.003898 0.007663 0.006007 0.003433 0.009267 0.008950 > 0.000680 > weight-l: 0.009245 > weight-t: 0.000169 0.000468 0.009222 0.002168 -0.010274 > weight-w: -0.928554 > ERROR:Expected at least 5 tokens per emtry in 'ttable-file', but only > found 4 > > > With the new moses we trained a simple model on a small corpus of 2000, > but during tuning, running the command: > > > /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108/training/mert-moses > -new.pl \ > input \ > reference \ > /MT/tools/moses/moses-cmd/src/mosesmt \ > ../model/moses.ini \ > --working-dir . \ > --rootdir /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108 > \ --mertdir /MT/tools/moses/mert > /MT/tools/scripts/reuse-weights.perl moses.ini > > we get: > > ... > Translating: l' efficacia della soluzione iniettabile è stata > confrontata con placebo su un arco di due ore di tempo , in due studi > che hanno coinvolto 805 pazienti schizofrenici o con condizioni > collegate , in stato di agitazione . > > mosesmt: ngramcache.cpp:87: int ngramcache::add(const int*, const > char*): Assertion `found == entry' failed. > Translating line 8 in thread id -572839664 > sh: line 1: 7365 Aborted > /MT/tools/moses/moses-cmd/src/mosesmt -config filtered/moses.ini > -inputtype 0 -w 0.00 -lm 0.11 -d 0.11 0.11 0.11 > 0.11 0.11 0.11 0.11 -tm 0.03 0.02 0.03 > 0.02 0.00 -n-best-list run1.best100.out 100 -i > /MT/it-en/tmp/tuning/input > run1.out > Exit code: 134 > The decoder died. CONFIG WAS -w 0.00 -lm 0.11 -d 0.11 > 0.11 0.11 0.11 0.11 0.11 0.11 -tm 0.03 > 0.02 0.03 0.02 0.00 > make: *** [moses.ini] Error 134 > > Thank you. > > -- Beppe > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] moses may 10
We are running into problems with the version of moses released on April 26th. mosesmt fails when using tables prouced with a previous version: Defined parameters (per moses.ini or switch): config: moses.ini distortion-file: 0-0 msd-bidirectional-fe 6 combinedBaseline/workdir/model/reorderTable/reordering-table distortion-limit: 6 input-factors: 0 lmodel-file: 1 0 5 baseline/languageModel/english.blm.mm mapping: 0 T 0 ttable-file: 0 0 5 combinedBaseline/workdir/model/phraseTable/phrase-table ttable-limit: 20 weight-d: 0.003898 0.007663 0.006007 0.003433 0.009267 0.008950 0.000680 weight-l: 0.009245 weight-t: 0.000169 0.000468 0.009222 0.002168 -0.010274 weight-w: -0.928554 ERROR:Expected at least 5 tokens per emtry in 'ttable-file', but only found 4 With the new moses we trained a simple model on a small corpus of 2000, but during tuning, running the command: /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108/training/mert-moses-new.pl \ input \ reference \ /MT/tools/moses/moses-cmd/src/mosesmt \ ../model/moses.ini \ --working-dir . \ --rootdir /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108 \ --mertdir /MT/tools/moses/mert /MT/tools/scripts/reuse-weights.perl moses.ini we get: ... Translating: l' efficacia della soluzione iniettabile è stata confrontata con placebo su un arco di due ore di tempo , in due studi che hanno coinvolto 805 pazienti schizofrenici o con condizioni collegate , in stato di agitazione . mosesmt: ngramcache.cpp:87: int ngramcache::add(const int*, const char*): Assertion `found == entry' failed. Translating line 8 in thread id -572839664 sh: line 1: 7365 Aborted /MT/tools/moses/moses-cmd/src/mosesmt -config filtered/moses.ini -inputtype 0 -w 0.00 -lm 0.11 -d 0.11 0.11 0.11 0.11 0.11 0.11 0.11 -tm 0.03 0.02 0.03 0.02 0.00 -n-best-list run1.best100.out 100 -i /MT/it-en/tmp/tuning/input > run1.out Exit code: 134 The decoder died. CONFIG WAS -w 0.00 -lm 0.11 -d 0.11 0.11 0.11 0.11 0.11 0.11 0.11 -tm 0.03 0.02 0.03 0.02 0.00 make: *** [moses.ini] Error 134 Thank you. -- Beppe ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
