[Moses-support] Incremental training issue

Hi, I have trained a SMT model using Moses on my own data. My goal is to build an incremental model so I can later on add more data. I have followed the instructions in Moses web page about incremental training. My data is preprocessed and prepared as it says. However, when trying to update and com

[Moses-support] Incremental Training

Dear Hieu, I just need a clear steps on how to do incremental training in moses, as the illustration in the manual is not cleared enough Thanks ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Incremental training

I compile my own xmlrpc-c library and executed these commands before running make: export LD_LIBRARY_PATH=/home/hieu/workspace/xmlrpc-c/xmlrpc-c-1.39.08/lib export LIBRARY_PATH=/home/hieu/workspace/xmlrpc-c/xmlrpc-c-1.39.08/lib export CPLUS_INCLUDE_PATH=/home/hieu/workspace/xmlrpc-c/xmlrpc

[Moses-support] Incremental training

Hello, I have been trying to set-up for incremental training [1], but fail on the preparation of compiling modified GIZA++, which apparently returns errors on xmlrpc-c library inclusion calls (doesn't find xmlrplc-c/base.hpp and others). I have tried setting it up on Ubuntu 14.04 and 16.04 with xm

[Moses-support] Incremental training using Mgiza

Hi all, I am using Mgiza to do incremental training from EN to DE. I followed the instruction in www.kyloo.net/software/doku.php/mgiza:forcealignment. I stuck at running the Model 4 alignment. In the instruction, the command is ./mgiza giza.en-ch/en-ch.gizacfg -c en-ch.snt -o output/en-ch -s ch.p

[Moses-support] Incremental Training

Hello, I am currently trying to use the incremental training of moses. I am following the http://www.statmt.org/moses/?n=Advanced.Incremental#ntoc6 tutorial and in step 3 (Build binary files) when I run the following command: ~/mosesdecoder/bin/mmlex-build corpus en es -o corpus.en-es.lex it thr

Re: [Moses-support] Incremental Training

ot.com/2015/09/moses-support-digest-vol-107-issue-3.html Marwa N. Refaie Date: Wed, 20 Jan 2016 11:41:36 +0200 From: adelkhali...@gmail.com To: moses-support@mit.edu Subject: [Moses-support] Incremental Training I had installed moses v2.1.1 on Ubuntu. Do I can make incremental training? wh

[Moses-support] Incremental Training

I had installed moses v2.1.1 on Ubuntu. Do I can make incremental training? what is it requirement. and how can i do it (Steps and tools)? Thanks ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Incremental training - mmlex out of bounds error

Hi All, To try incremental mode, I’ve trained the europarl (fr en) corpus with the option “-final-alignment-model hmm” I obtain following files: In /train/giza.fr-en/: En-fr.Ahmm.5.gz En-fr.aahmm.5 En-fr.cooc En-fr.gizacfg En-fr.hhmm.5 En-fr.thmm.5 And in /train/giza.en-fr/ : Fr-en

Re: [Moses-support] Incremental training after align_new.sh

@mit.edu Subject: [Moses-support] Incremental training after align_new.sh Hello everyone   When I using align_new.sh for the incremental training, I can’t understand “append the forward alignments (fwd) generated by align_new.sh to the forward (fwd) alignments generated by full_train.sh” witch

[Moses-support] Incremental training after align_new.sh

Hello everyone When I using align_new.sh for the incremental training, I can’t understand “append the forward alignments (fwd) generated by align_new.sh to the forward (fwd) alignments generated by full_train.sh” witch is written in your mail. For me, the files generated by

Re: [Moses-support] Incremental training

working directory (if its >>>>>>>> small you are testing it on small data right ? ). I will see what >>>>>>>> the >>>>>>>> problem is. >>>>>>>> >>>>>>>> >>>>>>>>

Re: [Moses-support] Incremental training

KEN ID must be unique for each token, in line : >>>>>>>> 24 roi 2 >>>>>>>> TOKEN ID 24 has already been assigned to: roi >>>>>>>> >>>>>>>> I took only 500 sentences pairs for full_train.sh and it worked >>>

Re: [Moses-support] Incremental training

>>> I took only 10 sentences for incremental alignment_new.sh which >>>>>>> generated the error and I found 8054 lines in the >>>>>>> new_corpus/new_tgt_file.vcb >>>>>>> Is there any problem? Can you please help me on the same. >

Re: [Moses-support] Incremental training

: >>>>>> 24 roi 2 >>>>>> TOKEN ID 24 has already been assigned to: roi >>>>>> >>>>>> I took only 500 sentences pairs for full_train.sh and it worked fine >>>>>> with 758 lines in the corpus/tgt_filename.vcb

Re: [Moses-support] Incremental training

N ID 24 has already been assigned to: roi >>>>> >>>>> I took only 500 sentences pairs for full_train.sh and it worked fine >>>>> with 758 lines in the corpus/tgt_filename.vcb file >>>>> >>>>> I took only 10 sentences for incremental alignment_n

Re: [Moses-support] Incremental training

;>> >>>> I took only 10 sentences for incremental alignment_new.sh which >>>> generated the error and I found 8054 lines in the >>>> new_corpus/new_tgt_file.vcb >>>> Is there any problem? Can you please help me on the same. >>>> >>>&

Re: [Moses-support] Incremental training

gt; >>>> Dear Ihab. >>>> There is a python script that was there in the google drive folder in >>>> the first mail I sent you. >>>> Please replace the existing file with my copy. >>>> >>>> It has to work. >>>> &g

Re: [Moses-support] Incremental training

Samsung Mobile >> >> >> >> ---- Original message ---- >> From: Ihab Ramadan >> Date: 05/11/2014 00:54 (GMT+09:00) >> To: 'Raj Dabre' >> Cc: moses-support@mit.edu >> Subject: RE: [Moses-support] Incremental training >> >>

Re: [Moses-support] Incremental training

ease replace the existing file with my copy. > > It has to work. > > Regards. > > > Sent from Samsung Mobile > > > > Original message > From: Ihab Ramadan > Date: 05/11/2014 00:54 (GMT+09:00) > To: 'Raj Dabre' > Cc: moses-

Re: [Moses-support] Incremental training

7;Raj Dabre' Cc: moses-support@mit.edu Subject: RE: [Moses-support] Incremental training Dear Raj,Your point is clear and I try to follow the steps you mentioned but I stuck now in the align_new.sh script which gives me this errorreading vocabulary files Reading vocabulary file from:

Re: [Moses-support] Incremental training

: 29107 q-1 4 Do you have any idea what this error means? From: Raj Dabre [mailto:prajda...@gmail.com] Sent: Tuesday, November 4, 2014 12:06 PM To: i.rama...@saudisoft.com Cc: moses-support@mit.edu Subject: Re: [Moses-support] Incremental training Dear Ihab, Perhaps I should have mentioned much

Re: [Moses-support] Incremental training

e > beginning and did not use the existing files generated > > Thanks Raj it still a great work > > > > > > *From:* Raj Dabre [mailto:prajda...@gmail.com] > *Sent:* Thursday, October 30, 2014 4:54 PM > > *To:* i.rama...@saudisoft.com > *Cc:* moses-support@mit.edu

Re: [Moses-support] Incremental training

: [Moses-support] Incremental training Ahh i totally forgot that part. Sorry. PFA. Just place them in the folder where the shell scripts full_train.sh and align_new.sh are. Hopefully it should run now. Please let me know if you succeed. On Thu, Oct 30, 2014 at 11:44 PM, Ihab

Re: [Moses-support] Incremental training

efixDEBUG: LogParsing Arguments > > ERROR: Cannot open configuration file configgiza.rev! > > > > > > This two files does not exists > > should they be generated from the installation? > > How to get them? > > > > *From:* Raj Dabre [mailto:prajda...@g

Re: [Moses-support] Incremental training

exists should they be generated from the installation? How to get them? From: Raj Dabre [mailto:prajda...@gmail.com] Sent: Sunday, October 26, 2014 6:21 PM To: i.rama...@saudisoft.com Cc: moses-support@mit.edu Subject: Re: [Moses-support] Incremental training Hello Ihab, I would

Re: [Moses-support] Incremental training

Thanks Raj, I will try this From: Raj Dabre [mailto:prajda...@gmail.com] Sent: Sunday, October 26, 2014 6:21 PM To: i.rama...@saudisoft.com Cc: moses-support@mit.edu Subject: Re: [Moses-support] Incremental training Hello Ihab, I would suggest using mgiza++. http://www.kyloo.net

Re: [Moses-support] Incremental training

Hello Ihab, I would suggest using mgiza++. http://www.kyloo.net/software/doku.php/mgiza:overview It is very easy to use. I also wrote some scripts to make it easy for training. Visit the link below for my scripts. https://drive.google.com/folderview?id=0B2gN8qfxTTUoSU43OFBhZXpPZ3M&usp=sharing Us

[Moses-support] Incremental training

Dear All, I just need a clear steps on how to do incremental training in moses, as the illustration in the manual is not cleared enough Thanks Best Regards Ihab Ramadan| Senior Developer| Saudisoft - Egypt | Tel +2 02 330 320 37 Ext- 0 | Mob+201007570826 | Fax+

Re: [Moses-support] Incremental Training

Thanks Hieu I will try this From: Hieu Hoang [mailto:hieuho...@gmail.com] Sent: Thursday, September 4, 2014 10:55 AM To: i.rama...@saudisoft.com; moses-support@mit.edu Subject: Re: [Moses-support] Incremental Training there is. I haven't used it before but there is documentation abo

Re: [Moses-support] Incremental Training

there is. I haven't used it before but there is documentation about it here: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc37 On 03/09/14 12:35, Ihab Ramadan wrote: Dear all, I want to ask if there is an incremental training feature in Moses and if so how to do it thanks Bes

[Moses-support] Incremental Training

Dear all, I want to ask if there is an incremental training feature in Moses and if so how to do it thanks Best Regards Ihab Ramadan| Senior Developer| Saudisoft - Egypt | Tel +2 02 330 320 37 Ext- 0 | Mob+201007570826 | Fax+20233032036 | Follow us on

Re: [Moses-support] incremental training

Incremental training in Moses is based upon work we did a few years back: http://homepages.inf.ed.ac.uk/miles/papers/naacl10b.pdf Table 3 shows that there is essentially no quality difference between incremental training and standard GIZA++ training. incremental (re) training is a lot faster. M

Re: [Moses-support] incremental training

Thanks for that Barry. I wonder if the stats come out the same as retraining a new system from scratch. From: Barry Haddow [bhad...@staffmail.ed.ac.uk] Sent: 30 October 2013 09:26 To: Read, James C Cc: moses-support@mit.edu Subject: Re: [Moses-support

Re: [Moses-support] incremental training

You mean this? http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc33 On 29/10/13 22:24, Read, James C wrote: > Hi, > > does anybody know if Moses currently has support for incremental training? > > James > > ___ > Moses-support mailing list > Mose

[Moses-support] incremental training

Hi, does anybody know if Moses currently has support for incremental training? James ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Incremental training without using incremental GIZA

Hi, you do not need incremental GIZA++ for the baseline run, but you need to run it with the HMM alignment models as final step and store intermediate files (which you likely have not done). Here some information: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc33 -phi On Sat, Jul 27,

Re: [Moses-support] Incremental training without using incremental GIZA

Can I use incremental GIZA++ for the new lines, even though I didn't use it for the baseline? (does mgiza give me everything inc-giza needs?) If not, I like the idea of just running word alignment on the new lines. Would I need to update any files besides *.A3.final.gz for steps 3+ to run correctl

Re: [Moses-support] Incremental training without using incremental GIZA

Hi, you could just run word alignment on the 50,000 lines, but you will get better performance if you somehow leverage the baseline parallel corpus for word alignment. One way is incremental GIZA++, the other is re-run everything. You could also try some middle ground of including some of the ba

[Moses-support] Incremental training without using incremental GIZA

Hello, I have a large phrase-based translation system. Alignment was done with mgiza, and took a few weeks. I now have a small amount of extremely relevant new bitext (~50,000 lines) that I would like to use to augment the model, without having to retrain everything. The new data contains many imp

Re: [Moses-support] incremental training questions

Did you smooth the probabilities in the regular phrase table? this usually adds around 0.3 BLEU. On 15 February 2013 10:32, Mirkin, Shachar wrote: > Hi, > > ** ** > > I’ve been trying for a while to use incremental training, but I’m running > into quite a few issues. > > The updates se

Re: [Moses-support] incremental training questions

Hello, I haven't used suffix arrays lately, but in the past they were only used for the forward translation probabilities, and not for the backards ones (due to complexity issues). This can result in a loss of 1 point BLEU. Holger On 02/15/2013 11:32 AM, Mirkin, Shachar wrote: Hi, I've b

[Moses-support] incremental training questions

Hi, I've been trying for a while to use incremental training, but I'm running into quite a few issues. The updates seem to be working fine, as evident from updates containing OOVs, but the performance when using the dynamic suffix array is much inferior (several BLEU points) in comparison to us

Re: [Moses-support] Incremental training crashes in run-giza and run-giza-inverse

Hi Guchun, As You, I used inc-giza-pp to work on incremental training and I got an issue but I'm not sure it was the same.. I realized that step 1 output (=prepare data) of train-model.perl workflow is not compatible with inc-giza-pp input format, where

Re: [Moses-support] Incremental training crashes in run-giza and run-giza-inverse

I am having the same problem with exactly the same error, Regards, Salameh On Fri, Aug 31, 2012 at 3:28 AM, Guchun Zhang wrote: > Hi, > > I am trying out incremental training again. I pulled a recent version of > mosesdecoder. And as instructed, I downloaded the inc-giza-pp-read-only > directory

[Moses-support] Incremental training crashes in run-giza and run-giza-inverse

Hi, I am trying out incremental training again. I pulled a recent version of mosesdecoder. And as instructed, I downloaded the inc-giza-pp-read-only directory from http://code.google.com/p/inc-giza-pp/. I compiled the incremental giza and copied the compiled GIZA++, plain2snt.out and snt2cooc.out

[Moses-support] Incremental Training - Doubt

Hi all, I have installed the incremental giza software but there are some problems when I run it. I have a trained model on 1M sentences and I intend to incrementally update the model using a batch of 4000 sentences. But when I run GIZA++ it is giving some weird memory error. *** glibc detected

Re: [Moses-support] Incremental training

incremental training for Giza is distinct from incremental training for the language model. we have worked on both --see Abby Levenberg's PhD http://homepages.inf.ed.ac.uk/miles/phd-projects/levenberg.pdf the short answer is "yes", but I don't think the incremental LM code has migrated from Abby

[Moses-support] Incremental training

Dear all, I'm starting to use the incremental training and I was wondering if it updates the language model as well. If the answer is not, is it possible to update the language model without restarting Moses? Thanks a lot Marco ___ Moses-support mailing

Re: [Moses-support] Incremental training

Many thanks, Qin. On 20 January 2012 15:17, Qin Gao wrote: > You can resume training using mgiza++, but it does not use online EM, just > load the model and continue training. You can also use force alignment with > mgiza++. > > Instructions for force alignment is here: > > http://geek.kyloo.net

Re: [Moses-support] Incremental training

You can resume training using mgiza++, but it does not use online EM, just load the model and continue training. You can also use force alignment with mgiza++. Instructions for force alignment is here: http://geek.kyloo.net/software/doku.php/mgiza:forcealignment Hope it meets your need, but if n

Re: [Moses-support] Incremental training

I am using the incremental version of giza++, found on http://code.google.com/p/inc-giza-pp/. Does mgiza support incremental training? Guchun On 20 January 2012 14:17, Qin Gao wrote: > I may missed early mails in this thread, are you using giza++ or mgiza? > --Q > > > > On Fri, Jan 20, 2012 at

Re: [Moses-support] Incremental training

I may missed early mails in this thread, are you using giza++ or mgiza? --Q On Fri, Jan 20, 2012 at 9:11 AM, Guchun Zhang wrote: > Hi again, > > I got the config file done as suggested by the sample config file. > However, I received a few errors regarding the parameters set in the config > fil

Re: [Moses-support] Incremental training

Hi again, I got the config file done as suggested by the sample config file. However, I received a few errors regarding the parameters set in the config file. ERROR: parameter 'stepk' does not exist. ERROR: Unrecognized attribute :step_k: ERROR: parameter 'oldtrprbs' does not exist. ERROR: Unreco

Re: [Moses-support] Incremental training

unfortunately, you can only set those arguments by changing the code in the training script train-model.perl line 970 and 971 set them both to 0 On 20/01/2012 10:26, Qin Gao wrote: These are lexical translation and distortion models produced by previous training, and can be produced by remo

Re: [Moses-support] Incremental training

Cheers. Guchun On 20 January 2012 03:26, Qin Gao wrote: > These are lexical translation and distortion models produced by previous > training, and can be produced by removing -nodump and -onlyaldumps from > giza options in moses training scripts. > > For documentation of giza and mgiza paramete

Re: [Moses-support] Incremental training

These are lexical translation and distortion models produced by previous training, and can be produced by removing -nodump and -onlyaldumps from giza options in moses training scripts. For documentation of giza and mgiza parameters, you can refer to http://geek.kyloo.net/software/doku.php/mgiza:c

Re: [Moses-support] Incremental training

Hi, these are paths to the model files produced for the original (not-updated) model. -phi On Thu, Jan 19, 2012 at 9:36 AM, Guchun Zhang wrote: > Hi, > > I am trying out incremental training. I am stuck in the Update and Compute > Alignments stage. In the sample config file for GIZA++, I am no

[Moses-support] Incremental training

Hi, I am trying out incremental training. I am stuck in the Update and Compute Alignments stage. In the sample config file for GIZA++, I am not quite sure what are the following parameters oldTrPrbs: (old translation probabilities?) oldAlPrbs: (old alignment probabilities?) and where are the f

Re: [Moses-support] Incremental training... how?

Hi, 2011/11/25 Philipp Koehn : > Hi, > > check the FAQ: > http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc27 I think he is actually speaking of this page you mention. It explains all the data preparation, then at the very end of the section, part "update model", it just says: « Now tha

Re: [Moses-support] Incremental training... how?

Hi, check the FAQ: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc27 -phi 2011/11/21 Kádár Tamás (KTamas) : > Hi > > I'm experimenting with incremental training. The documentation says: > "Updating a running Moses instance is done via XML RPC". And, uh, > that's it. I can't find any o

[Moses-support] Incremental training... how?

Hi I'm experimenting with incremental training. The documentation says: "Updating a running Moses instance is done via XML RPC". And, uh, that's it. I can't find any other details about it. Could somebody help me? Thanks and best regards, Tamas ___ Mose

Re: [Moses-support] Incremental training for SMT

Hi all, Probably not the appropriate thread to bring this discussion, but since 'suffix arrays' are only discussed in the context of Incremental training in the documentation, I'd like to think of them as relevant to the discussion on Incremental training. 1. I've trained a batch model using the

Re: [Moses-support] Incremental training for SMT

Hi, 2011/10/6 Philipp Koehn : > Hi, > > for the Moses support on this, please take a look at: > http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc27 I add my voice here as that's quite an interesting topic! Thanks for the reading! :-) In this page, I can read this: "Note that at the mome

Re: [Moses-support] Incremental training for SMT

if you want to understand the ideas behind the incremental training implemented in Moses, read: Stream-based Translation Models for Statistical Machine Translation, Abby Levenberg, Chris Callison-Burch and Miles Osborne, NAACL 2010 http://aclweb.org/anthology-new/N/N10/N10-1062.pdf Miles On 6 Oc

Re: [Moses-support] Incremental training for SMT

Hi, for the Moses support on this, please take a look at: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc27 -phi 2011/10/6 Jesús González Rubio : > 2011/10/6 HOANG Cong Duy Vu >> >> Hi all, >> >> I am working on the problem that tries to develop a SMT system that can >> learn increme

Re: [Moses-support] Incremental training for SMT

2011/10/6 HOANG Cong Duy Vu > Hi all, > > I am working on the problem that tries to develop a SMT system that can > learn incrementally. The scenario is as follows: > > - A state-of-the-art SMT system tries to translate a source language > sentence from users. > - Users identify some translation

[Moses-support] Incremental training for SMT

Hi all, I am working on the problem that tries to develop a SMT system that can learn incrementally. The scenario is as follows: - A state-of-the-art SMT system tries to translate a source language sentence from users. - Users identify some translation errors in translated sentence and then give

[Moses-support] Incremental training

Hi everyone, When I doing some experiments on incremental training, I refer to "http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc27";. However, I don't understand it clearly. Has anybody done *incremental training*? Is there any more *detailed reference*? It will be perfect if someone c