Re: [Moses-support] giza-training-error-EMS
HI All; I am getting the giza and giza In-versed failed error on EMS moses toy data default that is given in example of EMS and same error with my own given data. using ubuntu 12.04 Error given at the end of STDERR is : ERROR: Giza did not produce the output file /home/legends/work/ems-toy/training/giza.1/en-fr.A3.final. Is your corpus clean (reasonably-sized sentences)? at /home/legends/tools/mosesdecoder-RELEASE-2.1/scripts/training/train-model.perl line 1178. Corpus is already cleaned by the initial steps of EMS. Please help me out in solving this issue.. THANK YOU On Monday, March 3, 2014 4:02 PM, nadeem khan nad_sta...@yahoo.com wrote: HI All; I am getting the giza failed error on EMS moses toy data default that is given in example of EMS. using ubuntu 12.04 please check out the attached screen shot and EMS step files attached.. Error is : ERROR: Giza did not produce the output file /home/legends/work/ems-toy/training/giza.1/en-fr.A3.final. Is your corpus clean (reasonably-sized sentences)? at /home/legends/tools/mosesdecoder-RELEASE-2.1/scripts/training/train-model.perl line 1178.___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] mosesv2-decoder-error
Hi all; I am getting this error while running decoder with alignment flags: FeatureFunction: UnknownWordPenalty0 start: 9 end: 9 line=PhraseDictionaryMemory input-factor=0 output-factor=0 path=/home/legends/work/hin-eng/f5/model/phrase-table.gz num-features=5 table-limit=20 FeatureFunction: PhraseDictionaryMemory0 start: 10 end: 14 Loading SRILM0 /home/legends/work/hin-eng/f5/lm/urd-eng.lm: line 4317: warning: non-zero probability for unk in closed-vocabulary LM Loading Distortion0 Loading LexicalReordering0 Loading table into memory...done. Loading WordPenalty0 Loading UnknownWordPenalty0 Loading PhraseDictionaryMemory0 Start loading text SCFG phrase table. Moses format : [8.000] seconds Reading /home/legends/work/hin-eng/f5/model/phrase-table.gz 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 Exception: moses/Word.cpp:112 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vectorunsigned int, const StringPiece, bool) threw StrayFactorException because `fit'. You have configured 1 factors but the word | contains factor delimiter | too many times. Please help out in fixing it. THANKS Regards Nadeem___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] about segmentation flag of decoder
Hi all; I want to know about the use of different flags of moses while decoding.. as I am getting this kind of translation output while ran the decoder.. there is no kind of segmentation of alignment information for words.. I am using coomand given below: Output: Collecting options took 0.000 seconds Search took 0.100 seconds BEST TRANSLATION: in early years care and education وی|UNK|UNK|UNK کیو|UNK|UNK|UNK level 3 or the [] [total=-209.805] -19.000, -12.000, -200.000, -1.191, 0.000, -1.044, -2.693, 0.000, -2.999, -51.832, -15.323, -20.042, -2.518, -5.393, 6.999 Translation took 0.100 seconds Finished translating decoder command is : moses-configwork/eng-urd/f4/model/moses.ini-input-filework/eng-urd/f4/eval/urd-eng-test.lw.en1work/eng-urd/f4/urd.out2work/eng-urd/f4/eval/tuned.decode.out I want to add segmentation or other reordering etc information to the output plz help out in that regards and also Can Someone please tell me about the actual alignments and reordering of words from Source to target. In simple words we know in SMT the probablities are involved but I want to know the deeper side of these reorderings and alignments like for example a source word abc translated to xyz how the alignment of this word being done when it occurs in a sentence and how it will get in proper order.. If someone elaborate all this with some kind of example THANKS ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] about-actual-reordering-alignments
Hi all; Can Someone please tell me about the actual alignments and reordering of words from Source to target. In simple words we know in SMT the probablities are involved but I want to know the deeper side of these reorderings and alignments like for example a source word abc translated to xyz how the alignment of this word being done when it occurs in a sentence and how it will get in proper order.. If someone elaborate all this with some kind of example THANKS ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] moses.ini config weights
Hi all; I want to know some details about the Moses weights that are generated after training step under moses.ini config file and also we use tuning step to improve those weights. Where how actually these weights are being used? and how decoder take those weights while translating a input sentence? Can someone elaborate the use of these weights with example. What if we modified those weights manually without using tuning? Regards nadeem khan___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] How to Improve the translation
Hi Hieu; We are not same persons, I just saw this question and then your reply about tokenization and want to know the main purpose of it in SMT and as your said good tokenization for any source language can improve the BLEU score that is why asked this question. Regards Nadeem Khan On Wednesday, January 15, 2014 6:21 AM, Lane Schwartz dowob...@gmail.com wrote: See also: Final Report of the 2009 Summer Camp for Applied Language Exploration http://www.cs.jhu.edu/~ccb/publications/scale-2009-report.pdf On Tue, Jan 14, 2014 at 5:23 AM, Hieu Hoang hieu.ho...@ed.ac.uk wrote: The easiest way to improve translation is to find more data. The 2nd easiest way is to improve the tokenization of your languages On 11 January 2014 07:29, Asad A.Malik asad_12...@yahoo.com wrote: Hi All, I've developed Urdu to English SMT using MOSES, and it is currently giving me BLEU score of 8. Now I wanted to improve its translation so that it gives me higher BLEU score. Regards Asad A.Malik ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] about Tuning in moses
I am not getting this definition Tuning refers to the process of finding the optimal weights for this linear model, where optimal weights are those which maximise translation performance on a small set of parallel sentences (the tuning set). As i asked before why do we need it as already doing training of the system, where these optimal weights actually used while decoding? can you please elobrate effect of it with some kind of example. Why there are maximum 25 iterations in MERT? is there anykind of science behind it? and all other questions remains same as in previous mail. THANK YOU On , nadeem khan nad_sta...@yahoo.com wrote: Hi; I am not getting this definition Tuning refers to the process of finding the optimal weights for this linear model, where optimal weights are those which maximise translation performance on a small set of parallel sentences (the tuning set). As i asked before why do we need it as already doing training of the system, where these optimal weights actually used while decoding? can you please elobrate effect of it with some kind of example. Why there are maximum 25 iterations in MERT? is there anykind of science behind it? and all other questions remains same as in previous mail. THANK YOU On Monday, January 6, 2014 4:16 AM, Philipp Koehn pko...@inf.ed.ac.uk wrote: Hi, these are good questions that should be easy to answer if you understand the purpose of tuning when building machine translation systems. You can find some information here: http://www.statmt.org/moses/?n=FactoredTraining.Tuning -phi On Fri, Jan 3, 2014 at 2:26 PM, nadeem khan nad_sta...@yahoo.com wrote: Hi all I have a few question about tuning step of moses SMT. 1. Why we need tuning of the system ? as We can decode without it then why do we need it? 2. What is reason behind getting optimized weights and where these weights are being used while decoding??? 3. Why corpus is needed for tuning and why we cant use training datatset or testset for tunning of the system??? THANK YOU ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] about Tuning in moses
Someone please answer these completely, THANKS Hi all I have a few question about tuning step of moses SMT. 1. Why we need tuning of the system ? as We can decode without it then why do we need it? 2. What is reason behind getting optimized weights and where these weights are being used while decoding??? 3. Why corpus is needed for tuning and why we cant use training datatset or testset for tunning of the system??? THANK YOU Reply, Reply All On Friday, January 3, 2014 7:26 PM, nadeem khan nad_sta...@yahoo.com wrote: Hi all I have a few question about tuning step of moses SMT. 1. Why we need tuning of the system ? as We can decode without it then why do we need it? 2. What is reason behind getting optimized weights and where these weights are being used while decoding??? 3. Why corpus is needed for tuning and why we cant use training datatset or testset for tunning of the system??? THANK YOU___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] about Tuning in moses
Hi all I have a few question about tuning step of moses SMT. 1. Why we need tuning of the system ? as We can decode without it then why do we need it? 2. What is reason behind getting optimized weights and where these weights are being used while decoding??? 3. Why corpus is needed for tuning and why we cant use training datatset or testset for tunning of the system??? THANK YOU___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] about testing on part of training dataset
Hi Can anyone please tell me about why we got low BLEU score on a testset we get from training set for sparse resourced languages like Indian languages. I tested a Hindi system on both, means original testset and a testset extracted from Training data, just to get how things goes with testing the system some segments from training. But I got shocked when go through BLEU scores there is only 20-30% improvments in BLEU from original to training set. I was expecting a 100% BLEU from testset extracted from training data for experimenting. Kindly anyone of you tell me with some example why we cannot get 100% BLEU on testset from training ??? THANK YOU Regards Nadeem Khan___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS-Decoder-Problem
Hi Barry; I checked all the input steps files generated by EMS but cannot find any thing wrong there anywhere not in sgm file names nor the formatting of those ref and src files. Cannot getting it as its just not working on my 800 above testset otherwise whole experiment done successfully when using the fr-eng toy data set. Regards Nadeem On Tuesday, December 10, 2013 9:52 PM, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Nadeem It looks like something went wrong earlier in the EVALUATION section, possibly in the input-from-sgm step. I would check all the steps in this section for errors. It is also not clear to me that the truecaser will work with Hindi as it is designed for languages written in the latin script, cheers - Barry On 07/12/13 18:51, nadeem khan wrote: Hello Sir; I am using EMS now and getting into a problem with my data of hindi language. I ran EMS on config.toy just fine there was not a single error but when it comes to my own data and experiment I am getting stuck with BLEU and BLEU-c Crashed. When I invistaged the problem there is only 1 single Input Segment in test.input.tc.1. why and how the EMS taking only 1 segment from my input test-src.sgm file? and when I investigated further there is a fatal error under EVALUATION_test_nist-bleu-c.1.STDERR of no id in srcset. why I am getting that as I am giving it the complete sgm frame for wrapping out the output. I am sending you my those testdata sgm file as well as the input and output generated by EMS for my dataset. Please have a look at it and Reply with your kind comments to resolve these issues Waiting for your kind response THANK YOU Regards nadeem ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] single-input-sentence-problem-EMS
I ran EMS on config.toy with toydata successfully all runs just fine there was not a single error but when it comes to my own data of hindi and experiment I am getting stuck with BLEU and BLEU-c Crashed and in input file produced by EMS have only 1 single sentence from my above 900 sentences. When I invistaged the problem there is only 1 single Input Segment in test.input.tc.1. why and how the EMS taking only 1 segment from my input test-src.sgm file? and when I investigated further there is a fatal error under EVALUATION_test_nist-bleu-c.1.STDERR of no id in srcset. why I am getting that as I am giving it the complete sgm frame for wrapping out the output. Help out please___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] error in installing srilm
Hi Arthhika ; you need to install tcl to compile SRILM successfully. run the command given below in terminal(Ubuntu): sudoapt-getinstalltcltcl-devcshgawk Regards Nadeem Khan On Monday, December 9, 2013 10:08 AM, rohit dholakia rdhol...@sfu.ca wrote: Hi Arththika, Have you looked at the FAQ ? http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html I had set up SRILM long ago, and we have tcl support anyway. But, if you don't have tcl in your path or on your system, you might want to look at A1) part d. hope that helps. On Sun, Dec 8, 2013 at 8:58 PM, Arththika Paramanathan arthiparamanat...@gmail.com wrote: Hi, I faced some problems in installing srilm. Can anyone help me?. I attached the error file. Thank you -- regards, P.Arththika ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
