[Moses-support] removing identical pairs in phrase table
Hi, I'm using Moses in dialogue setting, where both source and target language are in English.This results in parroting back of the input, so I need to remove phrase-pairs where one phrase is a substring of the other. That'll require editing the phrase table after GIZA++ but before Moses.Is there a way to automate this? If not, how can I access the phrase table generated by GIZA++ and run Moses on that after editing? I appreciate your time in advance.___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] unsolved) Moses eating up too much memory
I apologize since my previous question on the same topic didn't specify enough.I'm following the steps in http://www.statmt.org/moses/?n=Moses.Baseline exactly the same way,and when I try to run Moses with the following command:~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with -output-hypo-score option)it starts eating up RAM until there's no space and starts taking hard disk space as swap memory,while it says its reading from phrase table.It went up to 17GB of swap memory and pretty much stopped.I binarised the phrase table and reordering table, and the following are the sizes of the binarised files-rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx-rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45 phrase-table.binphr.srctree.wa-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45 phrase-table.binphr.srcvoc-rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45 phrase-table.binphr.tgtdata.wa-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45 phrase-table.binphr.tgtvoc-rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00 reordering-table.binlexr.idx-rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00 reordering-table.binlexr.srctree-rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00 reordering-table.binlexr.tgtdata-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00 reordering-table.binlexr.voc0-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00 reordering-table.binlexr.voc1I also attached moses.ini which I tried to run.I'm very sure that this is not the way it should be, but can't figure out what is wrong since I followed the tutorial above..(and it's worked before) Please let me know if there is any additional information I should provide.I sincerely thank you very much for your time. moses.ini Description: Binary data ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] multi-threaded tuning not working
Hi, I've done tuning without the multi-core option,and it took about 20 hours for 650,000 lines of input files, and 5000 lines of tuning reference.I tried tuning with the multi-core option as--decoder-flags=-threads 4but it doesn't initiate anything.I have macbook pro with 2.4 GHz i5 dual core.Is there any known issue with this option? Thank you for your help in advance. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Running moses fills up hard disk space..
Hello, I'm running Moses after binarising it,and while it's reading from phrase table, it rapidly fills up hard disk space until there is no more space..I had 21GB and it wasn't enough..Once I quit terminal, the space is freed again..I successfully ran Moses before, so I don't understand why this happens..Is this natural? If so, how much reserve space do I need?and If not, what could have gone wrong? Thank you very much for your help in advance. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] getting WER metrics
Hello,sorry to ask another question.. I've done getting BLEU score in the past following the baseline tutorial,but is there a way to also get WER given a reference text? Thank you very much for your help.___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] a few supposedly simple questions..
Dear support team, thank you for your previous reply which worked out for me.I have a few questions which I think should be simple but couldn't find relevant information on the website. 1) When you run Moses and type in a sentence, is there any way you could have the translation withthe corresponding probability? 2) Also when you run and type in a sentence, is there a way to have not just one translation,but N-best candidates? (preferably with corresponding probabilities, which was the first question..) 3) I've done getting BLEU score using moses, but is there a way to also get word error rate to a reference? 4) After cleaning process, moses shows the number of lines in input and output text files,but I noticed that number of lines in output file decreased about 5%,resulting in non-matching number of linesfor input and output.Looking at the translation results, it seems like it worked fine somehow, but it gets me concerned.Why is it, and does it affect the line-match of input-output and the training process? I truly appreciate your help in advance. best,Andrew ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] error on training
Hello, I'm a novice to Moses, so I'm following the Baseline System page (http://www.statmt.org/moses/?n=Moses.Baseline) and got stuck at Training the Translation System part.When I enter the command starting with nohup nice as in the instruction,I only get a file named training.out and no new directory.The training.out file says:Using SCRIPTS_ROOTDIR: /Users/ipadmin/Desktop/mosesdecoder/scriptsUsing single-thread GIZAERROR: use --f to specify foreign language at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 368. So I've then tried the new way in the Training page (http://www.statmt.org/moses/?n=FactoredTraining.HomePage) with modification on euro and the language:train-model.perl -root-dir . --corpus corpus/euro --f de --e enand I get errors as following:Using SCRIPTS_ROOTDIR: /Users/ipadmin/Desktop/mosesdecoder/scriptsUse of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 236.Use of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 237.Use of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 244.Use of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 245.Use of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 247.Using single-thread GIZAUse of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or string at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 335.ERROR: Cannot find mkcls, GIZA++/mgiza, snt2cooc.out/snt2cooc in .You MUST specify the parameter -external-bin-dir at /Users/ipadmin/Desktop/mosesdecoder/scripts/training/train-model.perl line 335. I'm working on MacOSX 10.7 have successfully installed Moses and its prerequisites, and Giza++. I know this might be a stupid-user problem, but I will greatly appreciate it if you could give me a help.Thank you for your time. Sincerely,Andrew ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] question
Hello, I'm currently following the baseline system page (http://www.statmt.org/moses/?n=Moses.Baseline) and currently at the Training the Translation system part.I executed the command as in the tutorial, but the process finished instantly instead of supposed 1~2 hours,and instead of working/train/model folder with moses.ini file in it,I have working/train/corpus folder with files named en.vcb.classes, fr.vcb.classes, en.vcb.classes.cats, fr.vcb.classes.cats, en.vcb, fr.vcb, en-fr-int-train.snt, fr-en-int-train.snt.(Btw, I had to repeat the command a few times to get these files. On the first run, there were only two files, and then next time two more, and so on.) So it seems like something is being done, but I'm not getting moses.ini file..I've attached my training.out in case it helps.. This might be a tedious question to look through, but please help me out..Thank you very much. sincerely,Andrew training.out Description: Binary data ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
