Re: [Moses-support] Data for building a factored model

2016-05-06 Thread Sašo Kuntaric
Hi all, Thank you Philipp for all the useful info, I will take a closer look at the mentioned scripts. I do have one follow-up question. Like I said, I really enjoyed working with the factored corpora in the example. How were those created? Is there a tool I can use to create similar ones? Best

Re: [Moses-support] Data for building a factored model

2016-05-05 Thread Philipp Koehn
Hi, life is easier with factored models, if you use the experiment.perl set-up, where you just have to specify the factor set-up and scripts that generate factors. These scripts take the tokenized text and replace each word with a factor (e.g., replace each word with the POS tag). The POS LM is

Re: [Moses-support] Data for building a factored model

2016-05-04 Thread Sašo Kuntaric
Hello again, I believe I can wrap my head around the theoretical part, but the English and German corpora in the Moses factored model tutorial ( http://www.statmt.org/moses/?n=Moses.FactoredTutorial) look beautifully factored, so my question is how were the original corpora processed? Was a

[Moses-support] Data for building a factored model

2016-05-02 Thread Sašo Kuntaric
Hi all, I am having some issues producing the corpora in the correct format for Moses to execute factored training. I am looking at the factored tutorial on the Moses website and I am wondering, how to get such consistent corpora for two languages. What tools are being used and can they be