Re: [Moses-support] help for reusing moses-chart in a different (dependency tree - to - string) MT system
Hi, Hieu Thank you for your quick reply :-). But then, if the decoder can handle any number of non-terminal, it means I still don't understand why the rules I am generating do not seem to be used properly by the decoder. I think I will try to make a minimal example of the problem I have and come back for more wisdom... Thanks again for the answers, Fabien On Fri, Nov 9, 2012 at 12:15 AM, Hieu Hoang hieuho...@gmail.com wrote: hi fabien On 08/11/2012 01:25, Fabien Cromières wrote: Dear Moses community, First, thanks for providing the Research community with such a nice open-source tool. Second, I have some rather involved questions about the moses-chart decoder for syntax-based MT. Hopefully, someone having a good familiarity with it can answer me. Thanks in advance! Short version of the questions: * Is there any support for Synchronized Tree Substitution Grammar in moses-chart? (or any trick to use such a grammar with the decoder). no, moses-chart only does scfg. The decoder accepts tree input for tree-to-string decoding. This can approximate stsg decoding. * Can moses-chart handle rules with more than 2 Non Terminals? yes, the decoding algorithm is CKY+. It can handle arbitrary number of non-terms. * Can moses-chart be directly given (in any way) a compact representation of a set of parse of an input sentence? (and then just do the remaining work of selecting the best parse) i believe so, but i've never tried it. However, you can't assign probabilities of the input parse though. It wasn't really designed for forest decoding, so i'm not sure if it will be any good. Longer, more detailed, version: I am trying to improve a tree-to-string MT system (it is actually tree-to-tree, but it will be easier for me to describe it as a tree-to-string system). And I was hoping I could somehow re-use part of the Moses toolchain. Basically, my system uses a dependency tree representation of the input sentence. For example: he -is - (a-boy) (not easy to represent trees with strings in a readable way;hopefully, this notation is intuitive enough). I then have some Synchronized Tree Substitution rules, eg. for English-French (although unlike normal TSG, here the target side will be a flat string). R1: Y- he | il R2: X- Y-is-Z| Y est Z R3: Z-a - boy| un garcon R4: Z-a -boy| un enfant R5: Z- V-boy| V garcon R6: V-a |un The rules are already extracted, selected and mapped to the input by my system (in other words, the source side parsing is already done). This mean I have a somehow compact representation of every possible alternatives derivations like this: R1-R2-(R3|R4|(R5-R6)) Each derivation gives a different target sentence. From there, the problem is extracting the best derivation/translation according to language model and other features. It is essentially possible to do that with cube pruning or other beam-search approaches. However it would be interesting for me to re-use the work done for the moses-chart implementation. My first idea was to try and convert my TSG rules to the rule format expected by moses-chart. Unfortunately, this way, I cannot tell moses-chart in any way that the source side has already been parsed. However, I was hoping that I could make it easy for moses-chart to do the source-side parsing. Here is how I did it. For each input sentence, I use the position of the words instead of the words (to reduce parsing ambiguity). For example, the input sentence: he is a boy becomes: 1 2 3 4 when I feed it to moses-chart. Then I generate a rule file by adapting my rules accordingly (note that each Non Terminal encode the source position it should match in its name, ensuring the possible derivations that can be found by moses-chart is the same as those given by the parsing done by my system): 1 [X][X1] ||| il [X][X1] ||| ||| ... [X][X1] 2 [X][X4] [X][TOP] ||| [X][X1] est [X][X4] [X][TOP] ||| 0-0 2-2 ||| 3 4 [X][X4] ||| un garcon [X][X4] ||| ||| ... 3 4 [X][X4] ||| un enfant [X][X4] ||| ||| ... [X][X3] 4 [X][X4] ||| [X][X3] garcon [X][X4] ||| 0-0 ||| ... 3 [X][X3] ||| un [X][X1] ||| ||| ... However, I quickly found this was not giving the expected result (actually, no translation is found most of the time). After analyzing a bit more, I think I found out the problem is that I am generating some rules with more than 2 non-terminals. moses-chart do not complain about them, but it does not seem to be able to use them properly. I know hiero-style parsing will not work well with rules having more than 2 NT, so I guess this is to be expected. However, since it is not explicitly mentioned anywhere, I wanted to confirm that. Also, has anyone any suggestion for doing what I want to do with moses-chart? (since I have already the source-side parsing figured out, I don't think there is theoretical problems with having more than 2 NT per rule in my case; however,
[Moses-support] Two errors during decode and tuning
Hi there, I hope you can help me out with two new problems after installing a new Moses commit. I have spent some time on them, but to no avail. The first occurred during decode. The command is # /home/guchun/Work/moses/moses-scripts/generic/moses-parallel.pl -decoder /home/guchun/Work/moses/mosesdecoder/bin/moses -config /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.ini.1 -input-file /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.input.tc.1 *--jobs 32* -decoder-parameters -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.output.2 -text-type test # The models are built using the EMS factored example. The error message is - Split (-aa) were not entirely translated outputN=0 inputN=36 outputfile=model.input.tc.1.split17159-aa.trans inputfile=model.input.tc.1.split17159-aa everything crashed, not trying to resubmit jobs Got interrupt or something failed. kill_all_and_quit qdel 17791 Executing: qdel 17791 Exit code: 1 Translation was not performed correctly or some of the submitted jobs died. qdel function was called for all submitted jobs - Using bin/moses -thread 32 works fine. The second error is during the first iteration of tuning with a corpus having 4 factors. The command is # /home/guchun/Work/moses/moses-scripts/training/mert-moses.pl/home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/input.tc.1 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/reference.tc.1 /home/guchun/Work/moses/mosesdecoder/bin/moses /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/moses.filtered.ini.2 --nbest 100 --working-dir /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/tmp.2 --decoder-flags --rootdir /home/guchun/Work/moses/moses-scripts -mertdir /home/guchun/Work/moses/mosesdecoder/bin --no-filter-phrase-table *--jobs 32* # The error message is - The decoder returns the scores in this order: d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g Mismatched lambdas. Decoder returned d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g, we expected d d d d d d d lm lm w tm tm tm tm tm g g tm tm tm tm tm g g tm tm tm tm tm at /home/guchun/Work/moses/moses-scripts/training/mert-moses.pl line 1155. - I never had these problems with a previous commit back in March. Any hints will be warmly welcomed. Many thanks, Guchun ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Two salaried PhD positions in Machine Translation at Sheffield University - Marie Curie ITN project EXPERT
Applications are invited for two 3-year Early Stage Researcher PhD positions in corpus-based approaches to machine translation. The positions have been awarded as part of the new EU FP7 Marie-Curie Initial raining Network EXPERT (http://clg.wlv.ac.uk/projects/EXPERT/), concerned with the exploitation of empirical approaches to machine translation, including statistical machine translation and example-based machine translation. The EXPERT project puts together researchers from six European universities (The University of Sheffield, Universidad de Malaga, Universitaet des Saarlandes, Dublin City University, Universiteit van Amesterdam, and University of Wolverhampton) and five translation services and technology providers (Pangeanic, Hermes and Translated, Celer Soluciones and WordFast). The two projects allocated to the University of Sheffield are (i) learning from human feedback, which is concerned about methods to collect and incorporate human feedback into translation systems to improve the quality of translations; and (ii) estimating the quality and confidence of corpus-based approaches to machine translation and the quality of translated texts. The work in both projects will focus on statistical and machine learning techniques to address the problems, but it will also involve aspects of human translation studies, based on a strong interaction with some of the partners in EXPERT. The research will be conducted as part of the Natural Language Processing research group in Sheffield, with opportunities for collaborative work with several of the leading universities and industrial partners in EXPERT, and with the expectation of a 6-month secondment to 2-3 partners in this project. Further information about the research group can be found at: http://nlp.dcs.shef.ac.uk The successful individual will be employed as a salaried researcher at the University of Sheffield, while studying towards his/her PhD. Salary rates and additional mobility and training allowances are as per the Marie Curie programme (see Annex 3 of this document http://ec.europa.eu/research/participants/portal/download?docId=31562, page 69). For employment in the UK, the reference figure for gross *salary is 51,062 euros/year, *plus additional contribution towards mobility and training. Please check Section III.3 of the following link for further information on the Marie Curie scheme regarding eligibility criteria: ftp://ftp.cordis.europa.eu/pub/fp7/docs/fp7-mga-annex3intramulti_en.pdf. In particular, candidates cannot have lived in the UK for more than 12 months in the 3 years immediately prior to their appointment. Applicants should hold a good honours degree in a relevant field of study (e.g. computer science, engineering, mathematics) and have experience in natural language processing, machine translation or related area. They should also have a solid background in mathematics/statistics and excellent programming skills (C/C++, Java, Python/Perl, etc.). See the job specification for more details on the expected profile: http://www. dcs.shef.ac.uk/~lucia/resources/EXPERT_Marie_Curie_Early_Stage_Researcher.pdf The post is fixed term for three years with a start date of January 2013 or soon after. Closing Date: 10th December 2012 For applications: http://www.shef.ac.uk/jobs, Search and apply for jobs, job reference number: *UOS005598* In addition to submitting applications to the University of Sheffield, candidates should apply via the website provided by project coordinator: http://clg.wlv.ac.uk/projects/EXPERT/. For informal enquiries about this job and department, contact: Dr Lucia Specia on l.spe...@sheffield.ac.uk ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Two errors during decode and tuning
Hi Guchun I've pushed a change that may fix the second error. Can you let me know, and if it doesn't fix it, can you send me your ini file to help with testing. For the first error, could you post one of the decoder log files to see why the decoder is crashing, cheers - Barry On 09/11/12 11:15, Guchun Zhang wrote: Hi there, I hope you can help me out with two new problems after installing a new Moses commit. I have spent some time on them, but to no avail. The first occurred during decode. The command is # /home/guchun/Work/moses/moses-scripts/generic/moses-parallel.pl http://moses-parallel.pl -decoder /home/guchun/Work/moses/mosesdecoder/bin/moses -config /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.ini.1 -input-file /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.input.tc.1 *--jobs 32* -decoder-parameters -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.output.2 -text-type test # The models are built using the EMS factored example. The error message is - Split (-aa) were not entirely translated outputN=0 inputN=36 outputfile=model.input.tc.1.split17159-aa.trans inputfile=model.input.tc.1.split17159-aa everything crashed, not trying to resubmit jobs Got interrupt or something failed. kill_all_and_quit qdel 17791 Executing: qdel 17791 Exit code: 1 Translation was not performed correctly or some of the submitted jobs died. qdel function was called for all submitted jobs - Using bin/moses -thread 32 works fine. The second error is during the first iteration of tuning with a corpus having 4 factors. The command is # /home/guchun/Work/moses/moses-scripts/training/mert-moses.pl http://mert-moses.pl /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/input.tc.1 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/reference.tc.1 /home/guchun/Work/moses/mosesdecoder/bin/moses /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/moses.filtered.ini.2 --nbest 100 --working-dir /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR/rb//tuning/tmp.2 --decoder-flags --rootdir /home/guchun/Work/moses/moses-scripts -mertdir /home/guchun/Work/moses/mosesdecoder/bin --no-filter-phrase-table *--jobs 32* # The error message is - The decoder returns the scores in this order: d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g Mismatched lambdas. Decoder returned d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g, we expected d d d d d d d lm lm w tm tm tm tm tm g g tm tm tm tm tm g g tm tm tm tm tm at /home/guchun/Work/moses/moses-scripts/training/mert-moses.pl http://mert-moses.pl line 1155. - I never had these problems with a previous commit back in March. Any hints will be warmly welcomed. Many thanks, Guchun ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] (Closed) Problem with multiple translation pre-tags in Moses input (XML Markup)
Hi Barry, Thanks for your help ! Interesting verbose option... In fact, looking more precisely at the Moses log, I discovered that the problem came from one of my own treatment in the chain... Sorry, this problem wasn't a Moses problem. Thanks for your time. Best regards, Etienne Le 08/11/2012 16:25, Barry Haddow a écrit : Hi Etienne This works fine for me. I'm using multiple translation tags in the same sentence, and there's no confusion in the outputs. Are you using a recent version of Moses? You could try running with -v 3 - this will give extremely verbose output - and it show you the xml tags being parsed, and then list the translation options. This may give a clue as to what is going wrong, cheers - Barry On 06/11/12 16:12, Etienne Monneret wrote: Hi all, I'm testing the possibility to pre-tag some expressions translation in input texts (-xml-input exclusive). With 1 tag, it's working well (the sentence is of no interest, just showing some expressions I have in my e-glossaries) : Il y a de l'term translation=*return air*air repris/term dans cette allée froide entre les salles de réunion. ==There are *return air* in this allée cold between the meeting rooms. But, with 2 tags, Moses is confusing, and takes twice the second one : Il y a de l'term translation=*return air*air repris/term dans cette term translation=*cold aisle*allée froide/term entre les salles de réunion. ==There are *cold aisle* in this *cold aisle* between the meeting rooms. Is this a known bug ? Is there something I'm doing wrong ? Is there a way to get the good result ? Thanks for your help. Best regards, Etienne PS : this is a re-post with the good registered email address. Sorry if this request finally comes twice... ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Two errors during decode and tuning
Thanks, Barry. The following is the output file of one of the qsub jobs: Linux ubuntu11 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Defined parameters (per moses.ini or switch): config: /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.ini.1 cube-pruning-pop-limit: 5000 distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.1/reordering-table.1.0-0.wbe-msd-bidirectional-fe distortion-limit: 6 input-factors: 0 input-file: model.input.tc.1.split16701-aa inputtype: 0 lmodel-file: 0 0 7 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//lm/model.lm.1 0 1 7 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//lm/model=pos.lm.1 mapping: 0 T 0 search-algorithm: 1 stack: 5000 ttable-file: 1 0 0,1 5 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.1/phrase-table.0-0,1.1.1 ttable-limit: 20 weight-d: 0.063647 0.0630623 0.0827805 0.0571843 0.0856819 0.0255637 0.0459574 weight-l: 0.139309 0.0196229 weight-t: 0.136273 0.0197609 0.021245 0.0725207 -0.0529247 weight-w: -0.114467 *ERROR:illegal switch: -t-text-type* exit status 1 exit status 0 I checked -t wasn't present in the config file but appeared in the command. A typo in mert-moses.pl? I will pull the change you made and try it out in a moment. Cheers, Guchun On 9 November 2012 12:15, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Guchun I've pushed a change that may fix the second error. Can you let me know, and if it doesn't fix it, can you send me your ini file to help with testing. For the first error, could you post one of the decoder log files to see why the decoder is crashing, cheers - Barry On 09/11/12 11:15, Guchun Zhang wrote: Hi there, I hope you can help me out with two new problems after installing a new Moses commit. I have spent some time on them, but to no avail. The first occurred during decode. The command is # /home/guchun/Work/moses/moses-**scripts/generic/moses-**parallel.plhttp://moses-parallel.pl http://moses-parallel.pl -decoder /home/guchun/Work/moses/**mosesdecoder/bin/moses -config /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR//evaluation/model.**filtered.ini.1 -input-file /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR//evaluation/model.**input.tc.1 *--jobs 32* -decoder-parameters -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR//evaluation/model.**output.2 -text-type test # The models are built using the EMS factored example. The error message is - Split (-aa) were not entirely translated outputN=0 inputN=36 outputfile=model.input.tc.1.**split17159-aa.trans inputfile=model.input.tc.1.**split17159-aa everything crashed, not trying to resubmit jobs Got interrupt or something failed. kill_all_and_quit qdel 17791 Executing: qdel 17791 Exit code: 1 Translation was not performed correctly or some of the submitted jobs died. qdel function was called for all submitted jobs - Using bin/moses -thread 32 works fine. The second error is during the first iteration of tuning with a corpus having 4 factors. The command is # /home/guchun/Work/moses/moses-**scripts/training/mert-moses.pl http://mert-moses.pl /home/guchun/Data/mt-files/** tests/exp-new-training/model/**EN_US-FR_FR/rb//tuning/input.**tc.1 /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR/rb//tuning/**reference.tc.1 /home/guchun/Work/moses/**mosesdecoder/bin/moses /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR/rb//tuning/moses.**filtered.ini.2 --nbest 100 --working-dir /home/guchun/Data/mt-files/**tests/exp-new-training/model/**EN_US-FR_FR/rb//tuning/tmp.2 --decoder-flags --rootdir /home/guchun/Work/moses/moses-**scripts -mertdir /home/guchun/Work/moses/**mosesdecoder/bin --no-filter-phrase-table *--jobs 32* # The error message is - The decoder returns the scores in this order: d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g Mismatched lambdas. Decoder returned d d d d d d d lm lm w tm tm tm tm tm tm tm tm tm tm tm tm tm tm tm g g g g, we expected d d d d d d d lm lm w tm tm tm tm tm g g tm tm tm tm tm g g tm tm tm tm tm at /home/guchun/Work/moses/moses-**scripts/training/mert-moses.pl http://mert-moses.pl line 1155. - I never had these problems with a previous commit back in March. Any hints will be warmly welcomed. Many thanks, Guchun __**_ Moses-support mailing list
Re: [Moses-support] Two errors during decode and tuning
Hi Guchun The -text-type problem should be fixed by a commit I made yesterday. I haven't tested SGE operation though, so can you try again with the latest EMS? cheers - Barry On 09/11/12 13:26, Guchun Zhang wrote: Thanks, Barry. The following is the output file of one of the qsub jobs: Linux ubuntu11 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Defined parameters (per moses.ini or switch): config: /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.ini.1 cube-pruning-pop-limit: 5000 distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.1/reordering-table.1.0-0.wbe-msd-bidirectional-fe distortion-limit: 6 input-factors: 0 input-file: model.input.tc.1.split16701-aa inputtype: 0 lmodel-file: 0 0 7 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//lm/model.lm.1 0 1 7 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//lm/model=pos.lm.1 mapping: 0 T 0 search-algorithm: 1 stack: 5000 ttable-file: 1 0 0,1 5 /home/guchun/Data/mt-files/tests/exp-new-training/model/EN_US-FR_FR//evaluation/model.filtered.1/phrase-table.0-0,1.1.1 ttable-limit: 20 weight-d: 0.063647 0.0630623 0.0827805 0.0571843 0.0856819 0.0255637 0.0459574 weight-l: 0.139309 0.0196229 weight-t: 0.136273 0.0197609 0.021245 0.0725207 -0.0529247 weight-w: -0.114467 *ERROR:illegal switch: -t-text-type* exit status 1 exit status 0 I checked -t wasn't present in the config file but appeared in the command. A typo in mert-moses.pl http://mert-moses.pl? I will pull the change you made and try it out in a moment. Cheers, Guchun On 9 November 2012 12:15, Barry Haddow bhad...@staffmail.ed.ac.uk mailto:bhad...@staffmail.ed.ac.uk wrote: Hi Guchun I've pushed a change that may fix the second error. Can you let me know, and if it doesn't fix it, can you send me your ini file to help with testing. For the first error, could you post one of the decoder log files to see why the decoder is crashing, cheers - Barry On 09/11/12 11:15, Guchun Zhang wrote: Hi there, I hope you can help me out with two new problems after installing a new Moses commit. I have spent some time on them, but to no avail. The first occurred during decode. The command is # /home/guchun/Work/moses/moses- scripts/generic/moses- parallel.pl http://moses-parallel.pl http://moses-parallel.pl -decoder /home/guchun/Work/moses/ mosesdecoder/bin/moses -config /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. filtered.ini.1 -input-file /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. input.tc.1 *--jobs 32* -decoder-parameters -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. output.2 -text-type test # The models are built using the EMS factored example. The error message is - Split (-aa) were not entirely translated outputN=0 inputN=36 outputfile=model.input.tc.1. split17159-aa.trans inputfile=model.input.tc.1. split17159-aa everything crashed, not trying to resubmit jobs Got interrupt or something failed. kill_all_and_quit qdel 17791 Executing: qdel 17791 Exit code: 1 Translation was not performed correctly or some of the submitted jobs died. qdel function was called for all submitted jobs - Using bin/moses -thread 32 works fine. The second error is during the first iteration of tuning with a corpus having 4 factors. The command is # /home/guchun/Work/moses/moses- scripts/training/mert-moses.pl http://mert-moses.pl http://mert-moses.pl /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/input. tc.1 /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/ reference.tc.1 /home/guchun/Work/moses/ mosesdecoder/bin/moses /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/moses. filtered.ini.2 --nbest 100 --working-dir /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/tmp.2 --decoder-flags --rootdir /home/guchun/Work/moses/moses- scripts
Re: [Moses-support] Two errors during decode and tuning
Hi Barry, I am doing it now. :-) Cheers, Guchun On 9 November 2012 13:35, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Guchun The -text-type problem should be fixed by a commit I made yesterday. I haven't tested SGE operation though, so can you try again with the latest EMS? cheers - Barry On 09/11/12 13:26, Guchun Zhang wrote: Thanks, Barry. The following is the output file of one of the qsub jobs: Linux ubuntu11 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Defined parameters (per moses.ini or switch): config: /home/guchun/Data/mt-files/** tests/exp-new-training/model/**EN_US-FR_FR//evaluation/model.**filtered.ini.1 cube-pruning-pop-limit: 5000 distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6 /home/guchun/Data/mt-files/**tests/exp-new-training/model/** EN_US-FR_FR//evaluation/model.**filtered.1/reordering-table.1.**0-0.wbe-msd-bidirectional-fe distortion-limit: 6 input-factors: 0 input-file: model.input.tc.1.split16701-aa inputtype: 0 lmodel-file: 0 0 7 /home/guchun/Data/mt-files/** tests/exp-new-training/model/**EN_US-FR_FR//lm/model.lm.1 0 1 7 /home/guchun/Data/mt-files/**tests/exp-new-training/model/**EN_US-FR_FR//lm/model=pos.lm.1 mapping: 0 T 0 search-algorithm: 1 stack: 5000 ttable-file: 1 0 0,1 5 /home/guchun/Data/mt-files/** tests/exp-new-training/model/**EN_US-FR_FR//evaluation/model.** filtered.1/phrase-table.0-0,1.**1.1 ttable-limit: 20 weight-d: 0.063647 0.0630623 0.0827805 0.0571843 0.0856819 0.0255637 0.0459574 weight-l: 0.139309 0.0196229 weight-t: 0.136273 0.0197609 0.021245 0.0725207 -0.0529247 weight-w: -0.114467 *ERROR:illegal switch: -t-text-type* exit status 1 exit status 0 I checked -t wasn't present in the config file but appeared in the command. A typo in mert-moses.pl http://mert-moses.pl? I will pull the change you made and try it out in a moment. Cheers, Guchun On 9 November 2012 12:15, Barry Haddow bhad...@staffmail.ed.ac.ukmailto: bhad...@staffmail.ed.**ac.uk bhad...@staffmail.ed.ac.uk wrote: Hi Guchun I've pushed a change that may fix the second error. Can you let me know, and if it doesn't fix it, can you send me your ini file to help with testing. For the first error, could you post one of the decoder log files to see why the decoder is crashing, cheers - Barry On 09/11/12 11:15, Guchun Zhang wrote: Hi there, I hope you can help me out with two new problems after installing a new Moses commit. I have spent some time on them, but to no avail. The first occurred during decode. The command is # /home/guchun/Work/moses/moses- scripts/generic/moses- parallel.pl http://moses-parallel.pl http://moses-parallel.pl -decoder /home/guchun/Work/moses/ mosesdecoder/bin/moses -config /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. filtered.ini.1 -input-file /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. input.tc.1 *--jobs 32* -decoder-parameters -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR//evaluation/model. output.2 -text-type test # The models are built using the EMS factored example. The error message is - Split (-aa) were not entirely translated outputN=0 inputN=36 outputfile=model.input.tc.1. split17159-aa.trans inputfile=model.input.tc.1. split17159-aa everything crashed, not trying to resubmit jobs Got interrupt or something failed. kill_all_and_quit qdel 17791 Executing: qdel 17791 Exit code: 1 Translation was not performed correctly or some of the submitted jobs died. qdel function was called for all submitted jobs - Using bin/moses -thread 32 works fine. The second error is during the first iteration of tuning with a corpus having 4 factors. The command is # /home/guchun/Work/moses/moses- scripts/training/mert-moses.pl http://mert-moses.pl http://mert-moses.pl /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/input. tc.1 /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/ reference.tc.1 /home/guchun/Work/moses/ mosesdecoder/bin/moses /home/guchun/Data/mt-files/ tests/exp-new-training/model/ EN_US-FR_FR/rb//tuning/moses. filtered.ini.2 --nbest 100 --working-dir
[Moses-support] Proposed change: gv to xdg-open in ems
Hi, The EMS script creates a ps file that gets updated, showing its progress. Currently, the script attempts to display this postscript file with gv (if it's installed), or with display (if gv is not installed). For me at least, the ps file looks skewed when I view it with gv. I was wondering if anyone would mind if I make the script first look for xdg-open, and attempt to open the file using it as the default. xdg-open attempts to open the file with whatever application is the user's default preference for that type of file. So if you have your desktop environment configured to open ps files with gv, then that's what it would do. I have mine set up to open ps files with okular, so that's what it uses for me when I type xdg-open foo.gv. Does anyone have any objections to this change? Thanks, Lane ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
