Re: [Moses-support] SLF to PLF - tests - moses crashing

2010-08-09 Thread Chris Dyer
Yes, that will do it. On Mon, Aug 9, 2010 at 4:13 PM, Sylvain Raybaud wrote: > On Monday 09 August 2010 20:43:53 Chris Dyer wrote: >> I'm sorry I haven't had an opportunity to look into this yet >> (hopefully later this evening). But, one thing that you need to do is >> make sure that your config

Re: [Moses-support] SLF to PLF - tests - moses crashing

2010-08-09 Thread Sylvain Raybaud
On Monday 09 August 2010 20:43:53 Chris Dyer wrote: > I'm sorry I haven't had an opportunity to look into this yet > (hopefully later this evening). But, one thing that you need to do is > make sure that your config file has an entry setting the maximum > phrase size to a very large number, which p

Re: [Moses-support] SLF to PLF - tests - moses crashing

2010-08-09 Thread Chris Dyer
I'm sorry I haven't had an opportunity to look into this yet (hopefully later this evening). But, one thing that you need to do is make sure that your config file has an entry setting the maximum phrase size to a very large number, which prevents some bad pruning from taking place that can lead to

[Moses-support] Second CFP: Workshop "Bringing MT to the User: Research on Integrating MT in the Translation Industry"

2010-08-09 Thread Венцислав Жечев (Ventsislav Zhechev)
SECOND CALL FOR PAPERS -- "Bringing MT to the User: Research on Integrating MT in the Translation Industry" Second Joint EM+/CNGL Workshop (JEC 2010) http://web.me.com/emcnglworkshop/JEC2010 -

Re: [Moses-support] For the BLEU/NIST evaluation

2010-08-09 Thread Philipp Koehn
Hi, scores will differ for different test sets. If you randomly sample the test set from you parallel data, they should be relatively similar, though. You can compute confidence intervals with bootstrap resampling, which will give you some indication how reliable a, say, 0.5, 1.0, or 2.0 point di

Re: [Moses-support] For the BLEU/NIST evaluation

2010-08-09 Thread Wenlong Yang
Hi all, Is there any body can help on this? There might be some mistakes during the training of the Moses engine 2 (used the source language model), but the criterion to select the sample data for BLEU/NIST score evaluation is still sth I want to know/make sure. Thanks so, Wenlong 2010/8/7, We

Re: [Moses-support] SLF to PLF - tests - moses crashing

2010-08-09 Thread Sylvain Raybaud
Hello Here is the latest version of the script, which replace noise information with epsilon transitions. Moses successfully translates very small "toy" lattice generated with this script but on a real world example (http://perso.crans.org/raybaud/realworld.slf.gz) Moses fails with glibc com

Re: [Moses-support] Handling unknown words in Moses

2010-08-09 Thread John Burger
Philipp Koehn wrote: > this is not correct - LM cost is in the future cost estimate. > Obviously, this is a rather low probability, depending > on if the language model was trained with open or > closed vocabulary. And also whether the word is unknown to the LM or not, yes? Typically there are

[Moses-support] Handling unknown words in Moses

2010-08-09 Thread Trong Nghia Hoang
Hi everyone,I would like to ask for some details on how Moses deals with the handling of unknown words. As I read from the tutorial, unknown words are copied verbatim to the output. However, it is not clear of how we deal with the distortion limit while copying unknown words to the output.The si

Re: [Moses-support] Handling unknown words in Moses

2010-08-09 Thread Philipp Koehn
Hi, this is not correct - LM cost is in the future cost estimate. Obviously, this is a rather low probability, depending on if the language model was trained with open or closed vocabulary. The reordering of unknown words does cause often some strange reordering, due to the fact that an unknown w

Re: [Moses-support] Handling unknown words in Moses

2010-08-09 Thread Alexander Fraser
It seems like even if this is correctly implemented, unknown words would be delayed until the edge of the window they are in, due to trying to avoid paying the high LM cost until the last minute. LM cost is not in the future cost, so hypotheses paying this LM cost should lose to hypotheses delaying

[Moses-support] Moses release

2010-08-09 Thread Hieu Hoang
Dear Mosers and Mosettes For your delectation and degustation, we are rolling out a new version of moses which adds a bunch of new features and fixed a few bugs too. Download at the usual location, or SVN update the trunk. https://sourceforge.net/projects/mosesdecoder/ This is probably the la

Re: [Moses-support] Handling unknown words in Moses

2010-08-09 Thread Hieu Hoang
make sure the scores of the unknown word (-100) is included in your calculation of future cost. On 09/08/2010 08:02, nghi...@comp.nus.edu.sg wrote: > Hi everyone, > > I would like to ask for some details on how Moses deals with the handling > of unknown words. As I read from the tutorial, unkno

Re: [Moses-support] Handling unknown words in Moses

2010-08-09 Thread Barry Haddow
> With the default setting of Moses (phrase-based, distance-based reordering > ...), the handling of unknown words will be postponed (as we penalize them > severely : -100) until the very end. > > Therefore, it is likely that some unknown words are dropped and won't > appear in the output (due to

[Moses-support] Handling unknown words in Moses

2010-08-09 Thread nghiaht
Hi everyone, I would like to ask for some details on how Moses deals with the handling of unknown words. As I read from the tutorial, unknown words are copied verbatim to the output. However, it is not clear of how we deal with the distortion limit while copying unknown words to the output. The s