[Moses-support] compact lexical reordering model crashes
Hi Marcin, hi list, I can't load compact lexical reordering models. I don't know if it's generally broken or if you need some more information, but here's all the info I have without digging around in the code. I created it with the following settings, and there were no errors: mosesdecoder/bin/processLexicalTableMin -in reordering-table.wbe-msd-bidirectional-fe.gz -out reordering-table.wbe-msd-bidirectional-fe -threads 12 The config looks like this: # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr and this is the relevant bit of stderr: ScoreProducer: Distortion start: 0 end: 1 ScoreProducer: WordPenalty start: 1 end: 2 ScoreProducer: !UnknownWordPenalty start: 2 end: 3 Loading lexical distortion models...have 1 models ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 end: 9 Creating lexical reordering... weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 Loading table into memory...terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check Aborted Using yesterday's git commit (35cb7e6). best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
Hi Rico, I will take a look today. Just, to be sure: are you using a 64-bit machine? W dniu 16.11.2012 10:36, Rico Sennrich pisze: Hi Marcin, hi list, I can't load compact lexical reordering models. I don't know if it's generally broken or if you need some more information, but here's all the info I have without digging around in the code. I created it with the following settings, and there were no errors: mosesdecoder/bin/processLexicalTableMin -in reordering-table.wbe-msd-bidirectional-fe.gz -out reordering-table.wbe-msd-bidirectional-fe -threads 12 The config looks like this: # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr and this is the relevant bit of stderr: ScoreProducer: Distortion start: 0 end: 1 ScoreProducer: WordPenalty start: 1 end: 2 ScoreProducer: !UnknownWordPenalty start: 2 end: 3 Loading lexical distortion models...have 1 models ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 end: 9 Creating lexical reordering... weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 Loading table into memory...terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check Aborted Using yesterday's git commit (35cb7e6). best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
On Fri, 2012-11-16 at 12:02 +0100, Marcin Junczys-Dowmunt wrote: Hi Rico, I will take a look today. Just, to be sure: are you using a 64-bit machine? yes: uname -r 2.6.35.14-106.fc14.x86_64 ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
Hi Rico You need to remove the .minlexr from the end of the reordering table path. Moses is checking for a path ending with .minlexr.minlexr, which doesn't exist, so it's trying to load a memory-based table. I was hitting the same problem, although now I get the error: Using compact phrase table tcmalloc: large alloc 13312782167009714176 bytes == (nil) @ when trying to load the compact reordering table, cheers - Barry On 16/11/12 09:36, Rico Sennrich wrote: Hi Marcin, hi list, I can't load compact lexical reordering models. I don't know if it's generally broken or if you need some more information, but here's all the info I have without digging around in the code. I created it with the following settings, and there were no errors: mosesdecoder/bin/processLexicalTableMin -in reordering-table.wbe-msd-bidirectional-fe.gz -out reordering-table.wbe-msd-bidirectional-fe -threads 12 The config looks like this: # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr and this is the relevant bit of stderr: ScoreProducer: Distortion start: 0 end: 1 ScoreProducer: WordPenalty start: 1 end: 2 ScoreProducer: !UnknownWordPenalty start: 2 end: 3 Loading lexical distortion models...have 1 models ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 end: 9 Creating lexical reordering... weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 Loading table into memory...terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check Aborted Using yesterday's git commit (35cb7e6). best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] compact phrase table: missing alignments
Hi again, my last problems with the compact tables was a configuration problem, but I think this one isn't. I'm working on stuff that relies on word alignment during decoding, but for some phrase pairs, targetPhrase-GetAlignTerm() is missing alignment pairs in compact models (everything is dandy with in-memory tables). I traced the problem back to the variable a in PhraseTableCreator::EncodeLine(), which stores the alignment. Somewhere between its being filled with (lines 736-742) and the call to EncodeAlignment(), alignments go missing. Specifically, this happens in EncodeTargetPhrasePREnc (lines 639-645). I have no idea what the purpose of this bit is, so I won't touch it. But it would be great to have the full alignment information during decoding. I can send you a sample phrase table (4kb) to reproduce the problem. best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: can you put the sample data somewhere public like dropbox or googgle drive. i'll take a look at it as well the table is on https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ the command I used to build the compact table: ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out sample_data -alignment-info -rankscore 0 -nscores 3 To reproduce the problem, add a check to assert that alignment.size() 0 in PhraseTableCreator::EncodeAlignment() . ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
Hi, this might actually be a consequence of the compression method I use. You can also check and recompact the phrase table with the following option -encoding None It will be larger by some 20-30%, but it should not be missing alignment points. If it still does then this is an implementation mistake I have to fix. I will check your data now. W dniu 16.11.2012 16:22, Rico Sennrich pisze: On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: can you put the sample data somewhere public like dropbox or googgle drive. i'll take a look at it as well the table is on https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ the command I used to build the compact table: ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out sample_data -alignment-info -rankscore 0 -nscores 3 To reproduce the problem, add a check to assert that alignment.size() 0 in PhraseTableCreator::EncodeAlignment() . ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
OK, confirmed. The alignment points do not vanish with -encoding None. But this still seems to be a bug, the more efficient compression should not lose that many alignment points, this should only happen very rarely if at all. W dniu 16.11.2012 16:27, Marcin Junczys-Dowmunt pisze: Hi, this might actually be a consequence of the compression method I use. You can also check and recompact the phrase table with the following option -encoding None It will be larger by some 20-30%, but it should not be missing alignment points. If it still does then this is an implementation mistake I have to fix. I will check your data now. W dniu 16.11.2012 16:22, Rico Sennrich pisze: On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: can you put the sample data somewhere public like dropbox or googgle drive. i'll take a look at it as well the table is on https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ the command I used to build the compact table: ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out sample_data -alignment-info -rankscore 0 -nscores 3 To reproduce the problem, add a check to assert that alignment.size() 0 in PhraseTableCreator::EncodeAlignment() . ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
OK. Fixed. Hieu messed up the alignment :P But in his defense, I was looking directly at it and did not see it for some time :) W dniu 16.11.2012 16:39, Marcin Junczys-Dowmunt pisze: OK, confirmed. The alignment points do not vanish with -encoding None. But this still seems to be a bug, the more efficient compression should not lose that many alignment points, this should only happen very rarely if at all. W dniu 16.11.2012 16:27, Marcin Junczys-Dowmunt pisze: Hi, this might actually be a consequence of the compression method I use. You can also check and recompact the phrase table with the following option -encoding None It will be larger by some 20-30%, but it should not be missing alignment points. If it still does then this is an implementation mistake I have to fix. I will check your data now. W dniu 16.11.2012 16:22, Rico Sennrich pisze: On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: can you put the sample data somewhere public like dropbox or googgle drive. i'll take a look at it as well the table is on https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ the command I used to build the compact table: ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out sample_data -alignment-info -rankscore 0 -nscores 3 To reproduce the problem, add a check to assert that alignment.size() 0 in PhraseTableCreator::EncodeAlignment() . ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] phrase-table.gz - 0(Zero) Bytes
Hi. We have tried the baseline example from the Moses website. However, during the Training the Translation System stage, and as we trace the process/logs of training.out we get the final set of lines. Finished Thu Nov 15 20:00:08 2012 (6.6) consolidating the two halves @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate ./model/phrase-table.half.f2e.gz ./model/phrase-table.half.e2f.gz /dev/stdout | gzip -c ./model/phrase-table.gz sh: 1: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate: not found Executing: rm -f ./model/phrase-table.half.* (7) learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.1) [no factors] learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.2) building tables @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/lexical-reordering-score ./model/extract.o.sorted.gz 0.5 ./model/reordering-table. --model wbe msd wbe-msd-bidirectional-fe Lexical Reordering Scorer scores lexical reordering models of several types (hierarchical, phrase-based and word-based-extraction (8) learn generation model @ Thu Nov 15 20:00:08 PHT 2012 no generation model requested, skipping step (9) create moses.ini @ Thu Nov 15 20:00:08 PHT 2012 The case is almost the same with our own training data. Upon cheking on the produced phrase-table.gz we get 0 bytes. Please advice... Thanks. Erick___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] How to set up Moses Online Demonstrations
Hello, I am a student and I want to set up Moses Online Demostrtions. But I don't know how to build it. I search it in your sit but I just find a little message about it. In Web Translation and Web Interface ,there is some information about that. But I also have problem. 1. How to use moses to set up a system just like google translation. 2.Does moses Support web function? Or I should write script to Handle the requst from web. 3.If moses support web function,how can I use it. would you give me detailed answers. Or give some document about it. Thank you very much. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] How to set up Moses Online Demonstrations
Hi there, you can run Moses in server mode which would make available its translation capabilities via XML-RPC; The Moses manual is available here: - http://www.statmt.org/moses/manual/manual.pdf See the section on Moses Server on page 116 for more information. There's also a Wiki page from Herve Saint-Amand on web translation: - http://www.statmt.org/moses/?n=Moses.WebTranslation In 2010, I released MT Server Land, which basically allows to run Moses in a web service fashion -- source code is available here: - https://github.com/cfedermann/mt-serverland Hope that helps, best regards, Christian -- Dipl.-Inf. Christian Federmann, Researcher, Language Technology Lab Office +1.09 -- Phone +49-681/857-75-5353, Fax +49-681/857-75-5338 DFKI GmbH, Campus D3 2, Stuhlsatzenhausweg 3, 66123 Saarbruecken http://www.dfki.de/~cfedermann --- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 --- On 16.11.2012, at 03:58, dmp wrote: Hello, I am a student and I want to set up Moses Online Demostrtions. But I don't know how to build it. I search it in your sit but I just find a little message about it. In Web Translation and Web Interface ,there is some information about that. But I also have problem. 1. How to use moses to set up a system just like google translation. 2.Does moses Support web function? Or I should write script to Handle the requst from web. 3.If moses support web function,how can I use it. would you give me detailed answers. Or give some document about it. Thank you very much. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] phrase-table.gz - 0(Zero) Bytes
Hi, there is this error message: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate This binary should have been compiled with all the other code. Please check you compilation for errors. -phi On Thu, Nov 15, 2012 at 11:06 PM, Don Erick Bonus dej_bo...@yahoo.com wrote: Hi. We have tried the baseline example from the Moses website. However, during the Training the Translation System stage, and as we trace the process/logs of training.out we get the final set of lines. Finished Thu Nov 15 20:00:08 2012 (6.6) consolidating the two halves @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate ./model/phrase-table.half.f2e.gz ./model/phrase-table.half.e2f.gz /dev/stdout | gzip -c ./model/phrase-table.gz sh: 1: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate: not found Executing: rm -f ./model/phrase-table.half.* (7) learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.1) [no factors] learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.2) building tables @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/lexical-reordering-score ./model/extract.o.sorted.gz 0.5 ./model/reordering-table. --model wbe msd wbe-msd-bidirectional-fe Lexical Reordering Scorer scores lexical reordering models of several types (hierarchical, phrase-based and word-based-extraction (8) learn generation model @ Thu Nov 15 20:00:08 PHT 2012 no generation model requested, skipping step (9) create moses.ini @ Thu Nov 15 20:00:08 PHT 2012 The case is almost the same with our own training data. Upon cheking on the produced phrase-table.gz we get 0 bytes. Please advice... Thanks. Erick ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
