Re: [Moses-support] Issues with Score data
Hi Ligia It looks like you're mixing the old and new mert implementations. If you are using the mert-moses-new script, then you need to specify the mert directory using --mertdir. It is normally the directory 'mert' inside the moses distribution. See http://www.statmt.org/moses/?n=FactoredTraining.Tuning regards Barry On Monday 10 May 2010 18:12:40 Ligia Batrinca wrote: Hello I am trying to tune my system ( baseline ) and I have been having the following error ( copied from my cmert.log file): Seeding random numbers with system clock Scorer config string: Using scorer regularisation strategy: none Using scorer regularisation window: 0 Using case preservation: 1 Using reference length strategy: closest Data::score_type BLEU Data::Scorer type from Scorer: BLEU BleuScorer: 9 ScoreData: number_of_scores: 9 Loading Data from: statscore.data and features.data loading feature data from features.data loading score data from statscore.data Data loaded : [0] seconds terminate called after throwing an instance of 'std::runtime_error' what(): Score data is empty /home/user/mtsw/trunk/scripts/training/cmert-0.5/enhanced-mert: line 47: 19558 Aborted $SCRIPTS_ROOTDIR/training/cmert-0.5/mert -d $size I don't understand why it says that the score data is empty. I have been trying to run my tunning.sh script for the last 3 days, everytime I ended up with errors after (max) 12h running. Thanks for your help Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Issues with Score data
Hello, Thank you for the reply. I checked the webpage you suggested and in my opinion I am writing the command correctly. This is it: perl trunk/scripts/training/mert-moses.pl tuning/input.lowercased tuning/reference.lowercased trunk/moses-cmd/src/moses baseline-system/model/moses.ini --working-dir /home/user/mtsw/tuning/mertdir --rootdir /home/user/mtsw/trunk/scripts --no-filter-phrase-table --nbest 100 log5.mert I am not using mert-moses-new.pl Thanks Regards, Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Issues with Score data
Hi Ligia I'm fairly sure that the debug output you posted was from new mert. Can you run $SCRIPTS_ROOTDIR/training/cmert-0.5/mert (with no arguments) and see what happens? regards Barry On Tuesday 11 May 2010 14:17:26 Ligia Batrinca wrote: Hello, Thank you for the reply. I checked the webpage you suggested and in my opinion I am writing the command correctly. This is it: perl trunk/scripts/training/mert-moses.pl tuning/input.lowercased tuning/reference.lowercased trunk/moses-cmd/src/moses baseline-system/model/moses.ini --working-dir /home/user/mtsw/tuning/mertdir --rootdir /home/user/mtsw/trunk/scripts --no-filter-phrase-table --nbest 100 log5.mert I am not using mert-moses-new.pl Thanks Regards, Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Issues with Score data
Hello I did as indicated and this is the output: /home/user/mtsw/trunk/scripts/training/cmert-0.5/mert usage: mert -d dimensions (mandatory ) [-n retry ntimes (default 1)] [-othe indexes to optimize(default all)] [-tthe optimizer(default powell)] [-rthe random seed (defaults to system clock) [--sctype|-s] the scorer type (default BLEU) [--scconfig|-c] configuration string passed to scorer [--scfile|-S] comma separated list of scorer data files (default score.data) [--ffile|-F] comma separated list of feature data files (default feature.data) [--ifile|-i] the starting point data file (default init.opt) [-v] verbose level [--help|-h] print this message and exit regards, Ligia On Tue, May 11, 2010 at 3:57 PM, Barry Haddow bhad...@inf.ed.ac.uk wrote: Hi Ligia I'm fairly sure that the debug output you posted was from new mert. Can you run $SCRIPTS_ROOTDIR/training/cmert-0.5/mert (with no arguments) and see what happens? regards Barry On Tuesday 11 May 2010 14:17:26 Ligia Batrinca wrote: Hello, Thank you for the reply. I checked the webpage you suggested and in my opinion I am writing the command correctly. This is it: perl trunk/scripts/training/mert-moses.pl tuning/input.lowercased tuning/reference.lowercased trunk/moses-cmd/src/moses baseline-system/model/moses.ini --working-dir /home/user/mtsw/tuning/mertdir --rootdir /home/user/mtsw/trunk/scripts --no-filter-phrase-table --nbest 100 log5.mert I am not using mert-moses-new.pl Thanks Regards, Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Issues with Score data
Hi Ligia Yes, that's new mert. I'm not sure how it got into that location. I would suggest that you run 'make release' in the scripts directory, update your SCRIPTS_ROOTDIR, and check that running mert gives you something like: [tantalus]bhaddow: $SCRIPTS_ROOTDIR/training/cmert-0.5/mert usage: mert -d dimensions [tantalus]bhaddow: You should then be able to run mert-moses.pl successfully, regards Barry On Tuesday 11 May 2010 15:04:10 Ligia Batrinca wrote: Hello I did as indicated and this is the output: /home/user/mtsw/trunk/scripts/training/cmert-0.5/mert usage: mert -d dimensions (mandatory ) [-n retry ntimes (default 1)] [-othe indexes to optimize(default all)] [-tthe optimizer(default powell)] [-rthe random seed (defaults to system clock) [--sctype|-s] the scorer type (default BLEU) [--scconfig|-c] configuration string passed to scorer [--scfile|-S] comma separated list of scorer data files (default score.data) [--ffile|-F] comma separated list of feature data files (default feature.data) [--ifile|-i] the starting point data file (default init.opt) [-v] verbose level [--help|-h] print this message and exit regards, Ligia On Tue, May 11, 2010 at 3:57 PM, Barry Haddow bhad...@inf.ed.ac.uk wrote: Hi Ligia I'm fairly sure that the debug output you posted was from new mert. Can you run $SCRIPTS_ROOTDIR/training/cmert-0.5/mert (with no arguments) and see what happens? regards Barry On Tuesday 11 May 2010 14:17:26 Ligia Batrinca wrote: Hello, Thank you for the reply. I checked the webpage you suggested and in my opinion I am writing the command correctly. This is it: perl trunk/scripts/training/mert-moses.pl tuning/input.lowercased tuning/reference.lowercased trunk/moses-cmd/src/moses baseline-system/model/moses.ini --working-dir /home/user/mtsw/tuning/mertdir --rootdir /home/user/mtsw/trunk/scripts --no-filter-phrase-table --nbest 100 log5.mert I am not using mert-moses-new.pl Thanks Regards, Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Issues with Score data
Thanks I got the output you said and I am running my tuning script. Now I need to find an explanation why all that location missundertsanding happened :). Thanks Regard, Ligia On Tue, May 11, 2010 at 4:13 PM, Barry Haddow bhad...@inf.ed.ac.uk wrote: Hi Ligia Yes, that's new mert. I'm not sure how it got into that location. I would suggest that you run 'make release' in the scripts directory, update your SCRIPTS_ROOTDIR, and check that running mert gives you something like: [tantalus]bhaddow: $SCRIPTS_ROOTDIR/training/cmert-0.5/mert usage: mert -d dimensions [tantalus]bhaddow: You should then be able to run mert-moses.pl successfully, regards Barry On Tuesday 11 May 2010 15:04:10 Ligia Batrinca wrote: Hello I did as indicated and this is the output: /home/user/mtsw/trunk/scripts/training/cmert-0.5/mert usage: mert -d dimensions (mandatory ) [-n retry ntimes (default 1)] [-othe indexes to optimize(default all)] [-tthe optimizer(default powell)] [-rthe random seed (defaults to system clock) [--sctype|-s] the scorer type (default BLEU) [--scconfig|-c] configuration string passed to scorer [--scfile|-S] comma separated list of scorer data files (default score.data) [--ffile|-F] comma separated list of feature data files (default feature.data) [--ifile|-i] the starting point data file (default init.opt) [-v] verbose level [--help|-h] print this message and exit regards, Ligia On Tue, May 11, 2010 at 3:57 PM, Barry Haddow bhad...@inf.ed.ac.uk wrote: Hi Ligia I'm fairly sure that the debug output you posted was from new mert. Can you run $SCRIPTS_ROOTDIR/training/cmert-0.5/mert (with no arguments) and see what happens? regards Barry On Tuesday 11 May 2010 14:17:26 Ligia Batrinca wrote: Hello, Thank you for the reply. I checked the webpage you suggested and in my opinion I am writing the command correctly. This is it: perl trunk/scripts/training/mert-moses.pl tuning/input.lowercased tuning/reference.lowercased trunk/moses-cmd/src/moses baseline-system/model/moses.ini --working-dir /home/user/mtsw/tuning/mertdir --rootdir /home/user/mtsw/trunk/scripts --no-filter-phrase-table --nbest 100 log5.mert I am not using mert-moses-new.pl Thanks Regards, Ligia ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] moses may 10
We are running into problems with the version of moses released on April 26th. mosesmt fails when using tables prouced with a previous version: Defined parameters (per moses.ini or switch): config: moses.ini distortion-file: 0-0 msd-bidirectional-fe 6 combinedBaseline/workdir/model/reorderTable/reordering-table distortion-limit: 6 input-factors: 0 lmodel-file: 1 0 5 baseline/languageModel/english.blm.mm mapping: 0 T 0 ttable-file: 0 0 5 combinedBaseline/workdir/model/phraseTable/phrase-table ttable-limit: 20 weight-d: 0.003898 0.007663 0.006007 0.003433 0.009267 0.008950 0.000680 weight-l: 0.009245 weight-t: 0.000169 0.000468 0.009222 0.002168 -0.010274 weight-w: -0.928554 ERROR:Expected at least 5 tokens per emtry in 'ttable-file', but only found 4 With the new moses we trained a simple model on a small corpus of 2000, but during tuning, running the command: /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108/training/mert-moses-new.pl \ input \ reference \ /MT/tools/moses/moses-cmd/src/mosesmt \ ../model/moses.ini \ --working-dir . \ --rootdir /MT/tools/moses/bin/moses-scripts/scripts-20100510-1108 \ --mertdir /MT/tools/moses/mert /MT/tools/scripts/reuse-weights.perl moses.ini we get: ... Translating: l' efficacia della soluzione iniettabile รจ stata confrontata con placebo su un arco di due ore di tempo , in due studi che hanno coinvolto 805 pazienti schizofrenici o con condizioni collegate , in stato di agitazione . mosesmt: ngramcache.cpp:87: int ngramcache::add(const int*, const char*): Assertion `found == entry' failed. Translating line 8 in thread id -572839664 sh: line 1: 7365 Aborted /MT/tools/moses/moses-cmd/src/mosesmt -config filtered/moses.ini -inputtype 0 -w 0.00 -lm 0.11 -d 0.11 0.11 0.11 0.11 0.11 0.11 0.11 -tm 0.03 0.02 0.03 0.02 0.00 -n-best-list run1.best100.out 100 -i /MT/it-en/tmp/tuning/input run1.out Exit code: 134 The decoder died. CONFIG WAS -w 0.00 -lm 0.11 -d 0.11 0.11 0.11 0.11 0.11 0.11 0.11 -tm 0.03 0.02 0.03 0.02 0.00 make: *** [moses.ini] Error 134 Thank you. -- Beppe ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
For my purposes, even a hard-coded assumption of 1, along with a more transparent error message if the model isn't found, would do. Does anybody actually decode with in-memory phrase tables in real life? (well, I suppose some people do...) /Christian On Tue, 11 May 2010, Barry Haddow wrote: Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
Maybe adding another option in moses would be an good idea, at least you don't need to modify the ini files. Just modify the commands would be easier i guess. Best regards! Jie Jiang CNGL, School of Computing, Dublin City University, Glasnevin, Dublin 9. Tel: +353 (0)1 700 6724 2010/5/11 Christian Hardmeier c...@rax.ch For my purposes, even a hard-coded assumption of 1, along with a more transparent error message if the model isn't found, would do. Does anybody actually decode with in-memory phrase tables in real life? (well, I suppose some people do...) /Christian On Tue, 11 May 2010, Barry Haddow wrote: Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
On 11 May 2010 17:33, Christian Hardmeier c...@rax.ch wrote: For my purposes, even a hard-coded assumption of 1, along with a more transparent error message if the model isn't found, would do. Does anybody actually decode with in-memory phrase tables in real life? (well, I suppose some people do...) Google and anyone who actually wants to do more than optimise against a fixed dev/test set You can't afford to filter the phrase table when dealing with any old translation request Miles /Christian On Tue, 11 May 2010, Barry Haddow wrote: Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses may 10
there is a error message from the decoder. ERROR:Expected at least 5 tokens per entries in 'ttable-file', but only found 4 however, it's lost in all the credits and param explanation. So I've moved those out of the way when there's an error so that we can see message properly chris - we can add a default=1 but talked to pwilliams about it, would need to keep the back compatiblity in the mert scripts other places too. Would just be a source of constant bugs. On 11/05/2010 17:28, Barry Haddow wrote: Maybe a more transparent error message would help? On Tuesday 11 May 2010 17:20:26 Hieu Hoang wrote: i thought about making it back-compatible but the code gets messy and error prone. Theres now 3 more phrase table - the text SCFG, binary SCFG, and the suffix array. So i thought it better to take the punch now and feel a short, sharp pain rather than let it linger. however, anyone wants to put back the old code to make it back comp, they're welcome to, as long as u look after it On 11/05/2010 17:04, Christian Hardmeier wrote: Hi, The first error that you give is because the format of the moses.ini file has changed. You need to add an extra digit at the beginning of the line that specifies the ttable-file. Add 0 for a memory-based ttable, and 1 for a binarised ttable. Is there a reason why we can't have backwards compatibility here? I'm a bit concerned about moving to the latest decoder version since it will require me to update the configuration file of each and every system I've ever trained, and then they won't work with the old decoders any more. Couldn't the decoder figure out on its own whether it should be 0 or 1 if the indication is missing, as it used to do? Cheers, Christian ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
