Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

similarly reading the WMT14 paper from UEDIN, If not mistaken I read :
35.9 in the matrix : http://matrix.statmt.org/systems/show/2106
31.76 for B1 best system on page 101 of : 
http://www.statmt.org/wmt14/pdf/W14-3309.pdf

Maybe I do not take the good information.

Le 09/08/2015 23:07, Vincent Nguyen a écrit :
> Still looking at WMT 11 in fact something looks weird to me :
> This table suggests that the Karlsruhe IT obtained 30.5 Bleu score for
> FR to EN : http://matrix.statmt.org/matrix/systems_list/1669
> But reading the paper http://www.statmt.org/wmt11/pdf/WMT45.pdf shows
> 28.34 as the final score
>
> I am trying not to focus too much on Bleu scores but this is my only
> reference to compare my experiments.
>
>
> Le 04/08/2015 17:28, Barry Haddow a écrit :
>> Hi Vincent
>>
>> If you are comparing to the results of WMT11, then you can look at the
>> system descriptions to see what the authors did. In fact it's worth
>> looking at the WMT14 descriptions (WMT15 will be available next month)
>> to see how state-of-the-art systems are built.
>>
>> For fr-en or en-fr, the first thing to look at is the data. There are
>> some large data sets released for WMT and you can get a good gain from
>> just crunching more data (monolingual and parallel). Unfortunately
>> this takes more resources (disk, cpu etc) so you may run into trouble
>> here.
>>
>> The hierarchical models are much bigger so yes you will need more
>> disk. For fr-en/en-fr it's probably not worth the extra effort,
>>
>> cheers - Barry
>>
>> On 04/08/15 15:58, Vincent Nguyen wrote:
>>> thanks for your insights.
>>>
>>> I am just stuck by the Bleu difference between my 26 and the 30 of
>>> WMT11, and some results of WMT14 close to 36 or even 39
>>>
>>> I am currently having trouble with hierarchical rule set instead of
>>> lexical reordering
>>> wondering if I will get better results but I have an error message
>>> filesystem root low disk space before it crashes.
>>> is this model taking more disk space in some ways ?
>>>
>>> I will next try to use more corpora of which in domain with my
>>> internal TMX
>>>
>>> thanks for your answers.
>>>
>>> Le 04/08/2015 16:02, Hieu Hoang a écrit :
 On 03/08/2015 13:00, Vincent Nguyen wrote:
> Hi,
>
> Just a heads up on some EMS results, to get your experienced opinions.
>
> Corpus: Europarlv7 + NC2010
> fr => en
> Evaluation NC2011.
>
> 1) IRSTLM vs KenLM is much slower for training / tuning.
 that sounds right. KenLM is also multithreaded, IRSTLM can only be
 used in single-threaded decoding.
> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with
> KenLM)
 true
> 3) Compact Mode is faster than onDisk with a short test (77
> segments 96
> seconds, vs 126 seconds)
 true
> 4) One last thing I do not understand though :
> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
> know since NC2010 is part of training, should not be relevant)
> I got roughly the same BLEU score. I would have expected a higher
> score
> with a test set inculded in the training corpus.
>
> makes sense ?
>
>
> Next steps :
> What path should I use to get better scores ? I read the 'optimize'
> section of the website which deals more with speed
> and of course I will appply all of this but I was interested in
> tips to
> get more quality if possible.
 look into domain adaptation if you have multiple training corpora,
 some of which is in-domain and some out-of-domain.

 Other than that, getting good bleu score is a research open question.

 Well done on getting this far
> Thanks
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

Thanks for your feedback Rico.
What I find it hard to understand is how just by changing a test set you 
can have a 2 BLEU points difference without any other change in the 
tuning / training parameters.
it's even 4 points for UEDIN 2014 


Le 10/08/2015 09:56, Rico Sennrich a écrit :
> Hi Vincent,
>
> the KIT paper reports scores on newstest2010 (and newstest2009) in their
> system description paper, while the matrix shows scores on newstest2011.
> The UEDIN WMT14 paper reports scores on newstest2012, newstest2013, and
> newstest2014 (it may admittedly be hard to see which is which:
> newstest2013 is the default in that paper). The reason why people report
> results on an older test set is that system development for a shared
> task happens without access to the test set to avoid overfitting to the
> task. Time and space permitting, some experiments are repeated on that
> year's test set for the system description (like in table 6 in the UEDIN
> paper).
>
> best wishes,
> Rico
>
> On 10/08/15 08:32, Vincent Nguyen wrote:
>> similarly reading the WMT14 paper from UEDIN, If not mistaken I read :
>> 35.9 in the matrix : http://matrix.statmt.org/systems/show/2106
>> 31.76 for B1 best system on page 101 of :
>> http://www.statmt.org/wmt14/pdf/W14-3309.pdf
>>
>> Maybe I do not take the good information.
>>
>> Le 09/08/2015 23:07, Vincent Nguyen a écrit :
>>> Still looking at WMT 11 in fact something looks weird to me :
>>> This table suggests that the Karlsruhe IT obtained 30.5 Bleu score for
>>> FR to EN : http://matrix.statmt.org/matrix/systems_list/1669
>>> But reading the paper http://www.statmt.org/wmt11/pdf/WMT45.pdf shows
>>> 28.34 as the final score
>>>
>>> I am trying not to focus too much on Bleu scores but this is my only
>>> reference to compare my experiments.
>>>
>>>
>>> Le 04/08/2015 17:28, Barry Haddow a écrit :
 Hi Vincent

 If you are comparing to the results of WMT11, then you can look at the
 system descriptions to see what the authors did. In fact it's worth
 looking at the WMT14 descriptions (WMT15 will be available next month)
 to see how state-of-the-art systems are built.

 For fr-en or en-fr, the first thing to look at is the data. There are
 some large data sets released for WMT and you can get a good gain from
 just crunching more data (monolingual and parallel). Unfortunately
 this takes more resources (disk, cpu etc) so you may run into trouble
 here.

 The hierarchical models are much bigger so yes you will need more
 disk. For fr-en/en-fr it's probably not worth the extra effort,

 cheers - Barry

 On 04/08/15 15:58, Vincent Nguyen wrote:
> thanks for your insights.
>
> I am just stuck by the Bleu difference between my 26 and the 30 of
> WMT11, and some results of WMT14 close to 36 or even 39
>
> I am currently having trouble with hierarchical rule set instead of
> lexical reordering
> wondering if I will get better results but I have an error message
> filesystem root low disk space before it crashes.
> is this model taking more disk space in some ways ?
>
> I will next try to use more corpora of which in domain with my
> internal TMX
>
> thanks for your answers.
>
> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>> On 03/08/2015 13:00, Vincent Nguyen wrote:
>>> Hi,
>>>
>>> Just a heads up on some EMS results, to get your experienced opinions.
>>>
>>> Corpus: Europarlv7 + NC2010
>>> fr => en
>>> Evaluation NC2011.
>>>
>>> 1) IRSTLM vs KenLM is much slower for training / tuning.
>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>> used in single-threaded decoding.
>>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with
>>> KenLM)
>> true
>>> 3) Compact Mode is faster than onDisk with a short test (77
>>> segments 96
>>> seconds, vs 126 seconds)
>> true
>>> 4) One last thing I do not understand though :
>>> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
>>> know since NC2010 is part of training, should not be relevant)
>>> I got roughly the same BLEU score. I would have expected a higher
>>> score
>>> with a test set inculded in the training corpus.
>>>
>>> makes sense ?
>>>
>>>
>>> Next steps :
>>> What path should I use to get better scores ? I read the 'optimize'
>>> section of the website which deals more with speed
>>> and of course I will appply all of this but I was interested in
>>> tips to
>>> get more quality if possible.
>> look into domain adaptation if you have multiple training corpora,
>> some of which is in-domain and some out-of-domain.
>>
>> Other than that, getting good bleu score is a research open question.
>>
>> Well done on getting this far
>>> Thanks
>>>
>>>
>>>
>>> _

Re: [Moses-support] EMS results - makes sense ?

[Rico Sennrich]

Hi Vincent,

the KIT paper reports scores on newstest2010 (and newstest2009) in their 
system description paper, while the matrix shows scores on newstest2011. 
The UEDIN WMT14 paper reports scores on newstest2012, newstest2013, and 
newstest2014 (it may admittedly be hard to see which is which: 
newstest2013 is the default in that paper). The reason why people report 
results on an older test set is that system development for a shared 
task happens without access to the test set to avoid overfitting to the 
task. Time and space permitting, some experiments are repeated on that 
year's test set for the system description (like in table 6 in the UEDIN 
paper).

best wishes,
Rico

On 10/08/15 08:32, Vincent Nguyen wrote:
> similarly reading the WMT14 paper from UEDIN, If not mistaken I read :
> 35.9 in the matrix : http://matrix.statmt.org/systems/show/2106
> 31.76 for B1 best system on page 101 of :
> http://www.statmt.org/wmt14/pdf/W14-3309.pdf
>
> Maybe I do not take the good information.
>
> Le 09/08/2015 23:07, Vincent Nguyen a écrit :
>> Still looking at WMT 11 in fact something looks weird to me :
>> This table suggests that the Karlsruhe IT obtained 30.5 Bleu score for
>> FR to EN : http://matrix.statmt.org/matrix/systems_list/1669
>> But reading the paper http://www.statmt.org/wmt11/pdf/WMT45.pdf shows
>> 28.34 as the final score
>>
>> I am trying not to focus too much on Bleu scores but this is my only
>> reference to compare my experiments.
>>
>>
>> Le 04/08/2015 17:28, Barry Haddow a écrit :
>>> Hi Vincent
>>>
>>> If you are comparing to the results of WMT11, then you can look at the
>>> system descriptions to see what the authors did. In fact it's worth
>>> looking at the WMT14 descriptions (WMT15 will be available next month)
>>> to see how state-of-the-art systems are built.
>>>
>>> For fr-en or en-fr, the first thing to look at is the data. There are
>>> some large data sets released for WMT and you can get a good gain from
>>> just crunching more data (monolingual and parallel). Unfortunately
>>> this takes more resources (disk, cpu etc) so you may run into trouble
>>> here.
>>>
>>> The hierarchical models are much bigger so yes you will need more
>>> disk. For fr-en/en-fr it's probably not worth the extra effort,
>>>
>>> cheers - Barry
>>>
>>> On 04/08/15 15:58, Vincent Nguyen wrote:
 thanks for your insights.

 I am just stuck by the Bleu difference between my 26 and the 30 of
 WMT11, and some results of WMT14 close to 36 or even 39

 I am currently having trouble with hierarchical rule set instead of
 lexical reordering
 wondering if I will get better results but I have an error message
 filesystem root low disk space before it crashes.
 is this model taking more disk space in some ways ?

 I will next try to use more corpora of which in domain with my
 internal TMX

 thanks for your answers.

 Le 04/08/2015 16:02, Hieu Hoang a écrit :
> On 03/08/2015 13:00, Vincent Nguyen wrote:
>> Hi,
>>
>> Just a heads up on some EMS results, to get your experienced opinions.
>>
>> Corpus: Europarlv7 + NC2010
>> fr => en
>> Evaluation NC2011.
>>
>> 1) IRSTLM vs KenLM is much slower for training / tuning.
> that sounds right. KenLM is also multithreaded, IRSTLM can only be
> used in single-threaded decoding.
>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with
>> KenLM)
> true
>> 3) Compact Mode is faster than onDisk with a short test (77
>> segments 96
>> seconds, vs 126 seconds)
> true
>> 4) One last thing I do not understand though :
>> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
>> know since NC2010 is part of training, should not be relevant)
>> I got roughly the same BLEU score. I would have expected a higher
>> score
>> with a test set inculded in the training corpus.
>>
>> makes sense ?
>>
>>
>> Next steps :
>> What path should I use to get better scores ? I read the 'optimize'
>> section of the website which deals more with speed
>> and of course I will appply all of this but I was interested in
>> tips to
>> get more quality if possible.
> look into domain adaptation if you have multiple training corpora,
> some of which is in-domain and some out-of-domain.
>
> Other than that, getting good bleu score is a research open question.
>
> Well done on getting this far
>> Thanks
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

>> ___
>> Moses-support mai

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

Still looking at WMT 11 in fact something looks weird to me :
This table suggests that the Karlsruhe IT obtained 30.5 Bleu score for 
FR to EN : http://matrix.statmt.org/matrix/systems_list/1669
But reading the paper http://www.statmt.org/wmt11/pdf/WMT45.pdf shows 
28.34 as the final score

I am trying not to focus too much on Bleu scores but this is my only 
reference to compare my experiments.


Le 04/08/2015 17:28, Barry Haddow a écrit :
> Hi Vincent
>
> If you are comparing to the results of WMT11, then you can look at the 
> system descriptions to see what the authors did. In fact it's worth 
> looking at the WMT14 descriptions (WMT15 will be available next month) 
> to see how state-of-the-art systems are built.
>
> For fr-en or en-fr, the first thing to look at is the data. There are 
> some large data sets released for WMT and you can get a good gain from 
> just crunching more data (monolingual and parallel). Unfortunately 
> this takes more resources (disk, cpu etc) so you may run into trouble 
> here.
>
> The hierarchical models are much bigger so yes you will need more 
> disk. For fr-en/en-fr it's probably not worth the extra effort,
>
> cheers - Barry
>
> On 04/08/15 15:58, Vincent Nguyen wrote:
>> thanks for your insights.
>>
>> I am just stuck by the Bleu difference between my 26 and the 30 of
>> WMT11, and some results of WMT14 close to 36 or even 39
>>
>> I am currently having trouble with hierarchical rule set instead of
>> lexical reordering
>> wondering if I will get better results but I have an error message
>> filesystem root low disk space before it crashes.
>> is this model taking more disk space in some ways ?
>>
>> I will next try to use more corpora of which in domain with my 
>> internal TMX
>>
>> thanks for your answers.
>>
>> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>>>
>>> On 03/08/2015 13:00, Vincent Nguyen wrote:
 Hi,

 Just a heads up on some EMS results, to get your experienced opinions.

 Corpus: Europarlv7 + NC2010
 fr => en
 Evaluation NC2011.

 1) IRSTLM vs KenLM is much slower for training / tuning.
>>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>>> used in single-threaded decoding.
 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
 KenLM)
>>> true
 3) Compact Mode is faster than onDisk with a short test (77 
 segments 96
 seconds, vs 126 seconds)
>>> true
 4) One last thing I do not understand though :
 For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
 know since NC2010 is part of training, should not be relevant)
 I got roughly the same BLEU score. I would have expected a higher 
 score
 with a test set inculded in the training corpus.

 makes sense ?


 Next steps :
 What path should I use to get better scores ? I read the 'optimize'
 section of the website which deals more with speed
 and of course I will appply all of this but I was interested in 
 tips to
 get more quality if possible.
>>> look into domain adaptation if you have multiple training corpora,
>>> some of which is in-domain and some out-of-domain.
>>>
>>> Other than that, getting good bleu score is a research open question.
>>>
>>> Well done on getting this far

 Thanks



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

ok just for info / the record
1 million
=> 5 GB for 1st job
=> 5 GB for 2nd job (inverse)
Since I have more Gigs it's fine, now my 8 cpus are the limit .

thanks

Le 06/08/2015 19:45, Philipp Koehn a écrit :

Hi,

I have no hard numbers for this - just give it a try and it needs too 
much RAM,

then reduce to 100,000.

-phi

On Thu, Aug 6, 2015 at 10:17 AM, Vincent Nguyen > wrote:



is there like a rough estimate for how many gigs of ram you need
for 1 million sentence pairs batches ?

Le 06/08/2015 18:00, Philipp Koehn a écrit :

Hi,

if you run into memory problems with fast align, you can
add the following in the [TRAINING] section:

fast-align-max-lines = 100

This will run fast-align in parts of 1 million sentence pairs.

-phi


On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow
mailto:bhad...@inf.ed.ac.uk>> wrote:

Hi Vincent

It's a SIGKILL. Probably means it ran out of memory.

I'd recommend fast_align for this data set. Even if you
manage to get it running with mgiza it will still take a week
or so.

Just add
fast-align-settings = "-d -o -v"
to the TRAINING section of ems, and make sure that fast_align
is in your external-bin-dir.

cheers - Barry


On 06/08/15 08:40, Vincent Nguyen wrote:


so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can
look at the system descriptions to see what the authors
did. In fact it's worth looking at the WMT14 descriptions
(WMT15 will be available next month) to see how
state-of-the-art systems are built.

For fr-en or en-fr, the first thing to look at is the data.
There are some large data sets released for WMT and you can
get a good gain from just crunching more data (monolingual
and parallel). Unfortunately this takes more resources
(disk, cpu etc) so you may run into trouble here.

The hierarchical models are much bigger so yes you will
need more disk. For fr-en/en-fr it's probably not worth the
extra effort,

cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and
the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set
instead of
lexical reordering
wondering if I will get better results but I have an error
message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain
with my internal TMX

thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your
experienced opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM
can only be
used in single-threaded decoding.

2) BLEU results are almost the same (25.7 with Irstlm,
26.14 with KenLM)

true

3) Compact Mode is faster than onDisk with a short test
(77 segments 96
seconds, vs 126 seconds)

true

Re: [Moses-support] EMS results - makes sense ?

[Philipp Koehn]

Hi,

I have no hard numbers for this - just give it a try and it needs too much
RAM,
then reduce to 100,000.

-phi

On Thu, Aug 6, 2015 at 10:17 AM, Vincent Nguyen  wrote:

>
> is there like a rough estimate for how many gigs of ram you need for 1
> million sentence pairs batches ?
>
> Le 06/08/2015 18:00, Philipp Koehn a écrit :
>
> Hi,
>
> if you run into memory problems with fast align, you can
> add the following in the [TRAINING] section:
>
> fast-align-max-lines = 100
>
> This will run fast-align in parts of 1 million sentence pairs.
>
> -phi
>
>
> On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow  wrote:
>
>> Hi Vincent
>>
>> It's a SIGKILL. Probably means it ran out of memory.
>>
>> I'd recommend fast_align for this data set. Even if you manage to get it
>> running with mgiza it will still take a week or so.
>>
>> Just add
>> fast-align-settings = "-d -o -v"
>> to the TRAINING section of ems, and make sure that fast_align is in your
>> external-bin-dir.
>>
>> cheers - Barry
>>
>>
>> On 06/08/15 08:40, Vincent Nguyen wrote:
>>
>>
>> so I dropped my hierarchical model since I got an error.
>> Switched back to the "more data" by adding the Giga FR EN source
>> but now another error pops un running Giza Inverse :
>>
>> Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
>> Using multi-thread GIZA
>> using gzip
>> (2) running giza @ Wed Aug  5 21:03:56 CEST 2015
>> (2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
>> Executing: mkdir -p /home/moses/working/training/giza-inverse.7
>> Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc
>> /home/moses/working/training/giza-inverse.7/fr-en.cooc
>> /home/moses/working/training/prepared.7/en.vcb
>> /home/moses/working/training/prepared.7/fr.vcb
>> /home/moses/working/training/prepared.7/fr-en-int-train.snt
>> line 1000
>> line 2000
>>
>> ...
>> line 6609000
>> line 661
>> ERROR: Execution of:
>> /home/moses/working/bin/training-tools/mgizapp/snt2cooc
>> /home/moses/working/training/giza-inverse.7/fr-en.cooc
>> /home/moses/working/training/prepared.7/en.vcb
>> /home/moses/working/training/prepared.7/fr.vcb
>> /home/moses/working/training/prepared.7/fr-en-int-train.snt
>>   died with signal 9, without coredump
>>
>>
>> any clue what signal 9 means ?
>>
>>
>>
>> Le 04/08/2015 17:28, Barry Haddow a écrit :
>>
>> Hi Vincent
>>
>> If you are comparing to the results of WMT11, then you can look at the
>> system descriptions to see what the authors did. In fact it's worth looking
>> at the WMT14 descriptions (WMT15 will be available next month) to see how
>> state-of-the-art systems are built.
>>
>> For fr-en or en-fr, the first thing to look at is the data. There are
>> some large data sets released for WMT and you can get a good gain from just
>> crunching more data (monolingual and parallel). Unfortunately this takes
>> more resources (disk, cpu etc) so you may run into trouble here.
>>
>> The hierarchical models are much bigger so yes you will need more disk.
>> For fr-en/en-fr it's probably not worth the extra effort,
>>
>> cheers - Barry
>>
>> On 04/08/15 15:58, Vincent Nguyen wrote:
>>
>> thanks for your insights.
>>
>> I am just stuck by the Bleu difference between my 26 and the 30 of
>> WMT11, and some results of WMT14 close to 36 or even 39
>>
>> I am currently having trouble with hierarchical rule set instead of
>> lexical reordering
>> wondering if I will get better results but I have an error message
>> filesystem root low disk space before it crashes.
>> is this model taking more disk space in some ways ?
>>
>> I will next try to use more corpora of which in domain with my internal
>> TMX
>>
>> thanks for your answers.
>>
>> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>>
>>
>> On 03/08/2015 13:00, Vincent Nguyen wrote:
>>
>> Hi,
>>
>> Just a heads up on some EMS results, to get your experienced opinions.
>>
>> Corpus: Europarlv7 + NC2010
>> fr => en
>> Evaluation NC2011.
>>
>> 1) IRSTLM vs KenLM is much slower for training / tuning.
>>
>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>> used in single-threaded decoding.
>>
>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)
>>
>> true
>>
>> 3) Compact Mode is faster than onDisk with a short test (77 segments 96
>> seconds, vs 126 seconds)
>>
>> true
>>
>> 4) One last thing I do not understand though :
>> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
>> know since NC2010 is part of training, should not be relevant)
>> I got roughly the same BLEU score. I would have expected a higher score
>> with a test set inculded in the training corpus.
>>
>> makes sense ?
>>
>>
>> Next steps :
>> What path should I use to get better scores ? I read the 'optimize'
>> section of the website which deals more with speed
>> and of course I will appply all of this but I was interested in tips to
>> get more quality if possible.
>>
>> look into domain adaptation if you have multiple training corpora,
>> some of which is in-

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

is there like a rough estimate for how many gigs of ram you need for 1 
million sentence pairs batches ?


Le 06/08/2015 18:00, Philipp Koehn a écrit :

Hi,

if you run into memory problems with fast align, you can
add the following in the [TRAINING] section:

fast-align-max-lines = 100

This will run fast-align in parts of 1 million sentence pairs.

-phi


On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow > wrote:


Hi Vincent

It's a SIGKILL. Probably means it ran out of memory.

I'd recommend fast_align for this data set. Even if you manage to
get it running with mgiza it will still take a week or so.

Just add
fast-align-settings = "-d -o -v"
to the TRAINING section of ems, and make sure that fast_align is
in your external-bin-dir.

cheers - Barry


On 06/08/15 08:40, Vincent Nguyen wrote:


so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look
at the system descriptions to see what the authors did. In fact
it's worth looking at the WMT14 descriptions (WMT15 will be
available next month) to see how state-of-the-art systems are
built.

For fr-en or en-fr, the first thing to look at is the data.
There are some large data sets released for WMT and you can get
a good gain from just crunching more data (monolingual and
parallel). Unfortunately this takes more resources (disk, cpu
etc) so you may run into trouble here.

The hierarchical models are much bigger so yes you will need
more disk. For fr-en/en-fr it's probably not worth the extra
effort,

cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set
instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my
internal TMX

thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced
opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can
only be
used in single-threaded decoding.

2) BLEU results are almost the same (25.7 with Irstlm, 26.14
with KenLM)

true

3) Compact Mode is faster than onDisk with a short test (77
segments 96
seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the
evaluation (I
know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a
higher score
with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the
'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested
in tips to
get more quality if possible.

look into domain adaptation if you have multiple training
corpora,
some of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open
question.

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

I have a bunch of
unicode non character u-fdd3 is illegal for open interchange
in the SDTERR of tokenize / clean with the Giga data set.
Should I just ignore these ? these lines are skipped ?

Le 06/08/2015 18:00, Philipp Koehn a écrit :

Hi,

if you run into memory problems with fast align, you can
add the following in the [TRAINING] section:

fast-align-max-lines = 100

This will run fast-align in parts of 1 million sentence pairs.

-phi


On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow > wrote:


Hi Vincent

It's a SIGKILL. Probably means it ran out of memory.

I'd recommend fast_align for this data set. Even if you manage to
get it running with mgiza it will still take a week or so.

Just add
fast-align-settings = "-d -o -v"
to the TRAINING section of ems, and make sure that fast_align is
in your external-bin-dir.

cheers - Barry


On 06/08/15 08:40, Vincent Nguyen wrote:


so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look
at the system descriptions to see what the authors did. In fact
it's worth looking at the WMT14 descriptions (WMT15 will be
available next month) to see how state-of-the-art systems are
built.

For fr-en or en-fr, the first thing to look at is the data.
There are some large data sets released for WMT and you can get
a good gain from just crunching more data (monolingual and
parallel). Unfortunately this takes more resources (disk, cpu
etc) so you may run into trouble here.

The hierarchical models are much bigger so yes you will need
more disk. For fr-en/en-fr it's probably not worth the extra
effort,

cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set
instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my
internal TMX

thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced
opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can
only be
used in single-threaded decoding.

2) BLEU results are almost the same (25.7 with Irstlm, 26.14
with KenLM)

true

3) Compact Mode is faster than onDisk with a short test (77
segments 96
seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the
evaluation (I
know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a
higher score
with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the
'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested
in tips to
get more quality if possible.

look into domain adaptation if you have multiple training
corpora,
some of which is in-domain and some out-of-

Re: [Moses-support] EMS results - makes sense ?

[Philipp Koehn]

Hi,

this feature was added last month.

You can restrict the number of parallel processes that
EMS runs with a switch, e.g., "--max-active 1" for at
most 1 job at a time.

-phi

On Thu, Aug 6, 2015 at 12:12 PM, Dingyuan Wang 
wrote:

> When is the 'fast-align-max-lines' added? That's convenient. I had to
> write a wrapper script before to limit the lines to process.
> Also, can I not run two direction's fast-aligns/mgizas in parallel? I have
> the memory to run one at a time, but not two. (I also wrote a wrapper
> script to block one.)
>
> > 2015年8月7日 00:01於 "Philipp Koehn" 寫道：
> >>
> >> Hi,
> >>
> >> if you run into memory problems with fast align, you can
> >> add the following in the [TRAINING] section:
> >>
> >> fast-align-max-lines = 100
> >>
> >> This will run fast-align in parts of 1 million sentence pairs.
> >>
> >> -phi
> >>
> >>
> >> On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow 
> wrote:
> >>>
> >>> Hi Vincent
> >>>
> >>> It's a SIGKILL. Probably means it ran out of memory.
> >>>
> >>> I'd recommend fast_align for this data set. Even if you manage to get
> it running with mgiza it will still take a week or so.
> >>>
> >>> Just add
> >>> fast-align-settings = "-d -o -v"
> >>> to the TRAINING section of ems, and make sure that fast_align is in
> your external-bin-dir.
> >>>
> >>> cheers - Barry
> >>>
> >>>
> >>> On 06/08/15 08:40, Vincent Nguyen wrote:
> 
> 
>  so I dropped my hierarchical model since I got an error.
>  Switched back to the "more data" by adding the Giga FR EN source
>  but now another error pops un running Giza Inverse :
> 
>  Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
>  Using multi-thread GIZA
>  using gzip
>  (2) running giza @ Wed Aug  5 21:03:56 CEST 2015
>  (2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
>  Executing: mkdir -p /home/moses/working/training/giza-inverse.7
>  Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc
> /home/moses/working/training/giza-inverse.7/fr-en.cooc
> /home/moses/working/training/prepared.7/en.vcb
> /home/moses/working/training/prepared.7/fr.vcb
> /home/moses/working/training/prepared.7/fr-en-int-train.snt
>  line 1000
>  line 2000
> 
>  ...
>  line 6609000
>  line 661
>  ERROR: Execution of:
> /home/moses/working/bin/training-tools/mgizapp/snt2cooc
> /home/moses/working/training/giza-inverse.7/fr-en.cooc
> /home/moses/working/training/prepared.7/en.vcb
> /home/moses/working/training/prepared.7/fr.vcb
> /home/moses/working/training/prepared.7/fr-en-int-train.snt
>    died with signal 9, without coredump
> 
> 
>  any clue what signal 9 means ?
> 
> 
> 
>  Le 04/08/2015 17:28, Barry Haddow a écrit :
> >
> > Hi Vincent
> >
> > If you are comparing to the results of WMT11, then you can look at
> the system descriptions to see what the authors did. In fact it's worth
> looking at the WMT14 descriptions (WMT15 will be available next month) to
> see how state-of-the-art systems are built.
> >
> > For fr-en or en-fr, the first thing to look at is the data. There
> are some large data sets released for WMT and you can get a good gain from
> just crunching more data (monolingual and parallel). Unfortunately this
> takes more resources (disk, cpu etc) so you may run into trouble here.
> >
> > The hierarchical models are much bigger so yes you will need more
> disk. For fr-en/en-fr it's probably not worth the extra effort,
> >
> > cheers - Barry
> >
> > On 04/08/15 15:58, Vincent Nguyen wrote:
> >>
> >> thanks for your insights.
> >>
> >> I am just stuck by the Bleu difference between my 26 and the 30 of
> >> WMT11, and some results of WMT14 close to 36 or even 39
> >>
> >> I am currently having trouble with hierarchical rule set instead of
> >> lexical reordering
> >> wondering if I will get better results but I have an error message
> >> filesystem root low disk space before it crashes.
> >> is this model taking more disk space in some ways ?
> >>
> >> I will next try to use more corpora of which in domain with my
> internal TMX
> >>
> >> thanks for your answers.
> >>
> >> Le 04/08/2015 16:02, Hieu Hoang a écrit :
> >>>
> >>>
> >>> On 03/08/2015 13:00, Vincent Nguyen wrote:
> 
>  Hi,
> 
>  Just a heads up on some EMS results, to get your experienced
> opinions.
> 
>  Corpus: Europarlv7 + NC2010
>  fr => en
>  Evaluation NC2011.
> 
>  1) IRSTLM vs KenLM is much slower for training / tuning.
> >>>
> >>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
> >>> used in single-threaded decoding.
> 
>  2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with
> KenLM)
> >>>
> >>> true
> 
>  3) Compact Mode is faster than 

Re: [Moses-support] EMS results - makes sense ?

[Dingyuan Wang]

When is the 'fast-align-max-lines' added? That's convenient. I had to write
a wrapper script before to limit the lines to process.
Also, can I not run two direction's fast-aligns/mgizas in parallel? I have
the memory to run one at a time, but not two. (I also wrote a wrapper
script to block one.)

> 2015年8月7日 00:01於 "Philipp Koehn" 寫道：
>>
>> Hi,
>>
>> if you run into memory problems with fast align, you can
>> add the following in the [TRAINING] section:
>>
>> fast-align-max-lines = 100
>>
>> This will run fast-align in parts of 1 million sentence pairs.
>>
>> -phi
>>
>>
>> On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow 
wrote:
>>>
>>> Hi Vincent
>>>
>>> It's a SIGKILL. Probably means it ran out of memory.
>>>
>>> I'd recommend fast_align for this data set. Even if you manage to get
it running with mgiza it will still take a week or so.
>>>
>>> Just add
>>> fast-align-settings = "-d -o -v"
>>> to the TRAINING section of ems, and make sure that fast_align is in
your external-bin-dir.
>>>
>>> cheers - Barry
>>>
>>>
>>> On 06/08/15 08:40, Vincent Nguyen wrote:


 so I dropped my hierarchical model since I got an error.
 Switched back to the "more data" by adding the Giga FR EN source
 but now another error pops un running Giza Inverse :

 Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
 Using multi-thread GIZA
 using gzip
 (2) running giza @ Wed Aug  5 21:03:56 CEST 2015
 (2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
 Executing: mkdir -p /home/moses/working/training/giza-inverse.7
 Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
 line 1000
 line 2000

 ...
 line 6609000
 line 661
 ERROR: Execution of:
/home/moses/working/bin/training-tools/mgizapp/snt2cooc
/home/moses/working/training/giza-inverse.7/fr-en.cooc
/home/moses/working/training/prepared.7/en.vcb
/home/moses/working/training/prepared.7/fr.vcb
/home/moses/working/training/prepared.7/fr-en-int-train.snt
   died with signal 9, without coredump


 any clue what signal 9 means ?



 Le 04/08/2015 17:28, Barry Haddow a écrit :
>
> Hi Vincent
>
> If you are comparing to the results of WMT11, then you can look at
the system descriptions to see what the authors did. In fact it's worth
looking at the WMT14 descriptions (WMT15 will be available next month) to
see how state-of-the-art systems are built.
>
> For fr-en or en-fr, the first thing to look at is the data. There are
some large data sets released for WMT and you can get a good gain from just
crunching more data (monolingual and parallel). Unfortunately this takes
more resources (disk, cpu etc) so you may run into trouble here.
>
> The hierarchical models are much bigger so yes you will need more
disk. For fr-en/en-fr it's probably not worth the extra effort,
>
> cheers - Barry
>
> On 04/08/15 15:58, Vincent Nguyen wrote:
>>
>> thanks for your insights.
>>
>> I am just stuck by the Bleu difference between my 26 and the 30 of
>> WMT11, and some results of WMT14 close to 36 or even 39
>>
>> I am currently having trouble with hierarchical rule set instead of
>> lexical reordering
>> wondering if I will get better results but I have an error message
>> filesystem root low disk space before it crashes.
>> is this model taking more disk space in some ways ?
>>
>> I will next try to use more corpora of which in domain with my
internal TMX
>>
>> thanks for your answers.
>>
>> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>>>
>>>
>>> On 03/08/2015 13:00, Vincent Nguyen wrote:

 Hi,

 Just a heads up on some EMS results, to get your experienced
opinions.

 Corpus: Europarlv7 + NC2010
 fr => en
 Evaluation NC2011.

 1) IRSTLM vs KenLM is much slower for training / tuning.
>>>
>>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>>> used in single-threaded decoding.

 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with
KenLM)
>>>
>>> true

 3) Compact Mode is faster than onDisk with a short test (77
segments 96
 seconds, vs 126 seconds)
>>>
>>> true

 4) One last thing I do not understand though :
 For sake of checking, I replaced NC2011 by NC2010 in the
evaluation (I
 know since NC2010 is part of training, should not be relevant)
 I got roughly the same BLEU score. I would have expected a higher
score
 with a test set inculded in the training corpus.

 makes sense ?


>>

Re: [Moses-support] EMS results - makes sense ?

[Philipp Koehn]

Hi,

if you run into memory problems with fast align, you can
add the following in the [TRAINING] section:

fast-align-max-lines = 100

This will run fast-align in parts of 1 million sentence pairs.

-phi


On Thu, Aug 6, 2015 at 7:28 AM, Barry Haddow  wrote:

> Hi Vincent
>
> It's a SIGKILL. Probably means it ran out of memory.
>
> I'd recommend fast_align for this data set. Even if you manage to get it
> running with mgiza it will still take a week or so.
>
> Just add
> fast-align-settings = "-d -o -v"
> to the TRAINING section of ems, and make sure that fast_align is in your
> external-bin-dir.
>
> cheers - Barry
>
>
> On 06/08/15 08:40, Vincent Nguyen wrote:
>
>
> so I dropped my hierarchical model since I got an error.
> Switched back to the "more data" by adding the Giga FR EN source
> but now another error pops un running Giza Inverse :
>
> Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
> Using multi-thread GIZA
> using gzip
> (2) running giza @ Wed Aug  5 21:03:56 CEST 2015
> (2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
> Executing: mkdir -p /home/moses/working/training/giza-inverse.7
> Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc
> /home/moses/working/training/giza-inverse.7/fr-en.cooc
> /home/moses/working/training/prepared.7/en.vcb
> /home/moses/working/training/prepared.7/fr.vcb
> /home/moses/working/training/prepared.7/fr-en-int-train.snt
> line 1000
> line 2000
>
> ...
> line 6609000
> line 661
> ERROR: Execution of:
> /home/moses/working/bin/training-tools/mgizapp/snt2cooc
> /home/moses/working/training/giza-inverse.7/fr-en.cooc
> /home/moses/working/training/prepared.7/en.vcb
> /home/moses/working/training/prepared.7/fr.vcb
> /home/moses/working/training/prepared.7/fr-en-int-train.snt
>   died with signal 9, without coredump
>
>
> any clue what signal 9 means ?
>
>
>
> Le 04/08/2015 17:28, Barry Haddow a écrit :
>
> Hi Vincent
>
> If you are comparing to the results of WMT11, then you can look at the
> system descriptions to see what the authors did. In fact it's worth looking
> at the WMT14 descriptions (WMT15 will be available next month) to see how
> state-of-the-art systems are built.
>
> For fr-en or en-fr, the first thing to look at is the data. There are some
> large data sets released for WMT and you can get a good gain from just
> crunching more data (monolingual and parallel). Unfortunately this takes
> more resources (disk, cpu etc) so you may run into trouble here.
>
> The hierarchical models are much bigger so yes you will need more disk.
> For fr-en/en-fr it's probably not worth the extra effort,
>
> cheers - Barry
>
> On 04/08/15 15:58, Vincent Nguyen wrote:
>
> thanks for your insights.
>
> I am just stuck by the Bleu difference between my 26 and the 30 of
> WMT11, and some results of WMT14 close to 36 or even 39
>
> I am currently having trouble with hierarchical rule set instead of
> lexical reordering
> wondering if I will get better results but I have an error message
> filesystem root low disk space before it crashes.
> is this model taking more disk space in some ways ?
>
> I will next try to use more corpora of which in domain with my internal
> TMX
>
> thanks for your answers.
>
> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>
>
> On 03/08/2015 13:00, Vincent Nguyen wrote:
>
> Hi,
>
> Just a heads up on some EMS results, to get your experienced opinions.
>
> Corpus: Europarlv7 + NC2010
> fr => en
> Evaluation NC2011.
>
> 1) IRSTLM vs KenLM is much slower for training / tuning.
>
> that sounds right. KenLM is also multithreaded, IRSTLM can only be
> used in single-threaded decoding.
>
> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)
>
> true
>
> 3) Compact Mode is faster than onDisk with a short test (77 segments 96
> seconds, vs 126 seconds)
>
> true
>
> 4) One last thing I do not understand though :
> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
> know since NC2010 is part of training, should not be relevant)
> I got roughly the same BLEU score. I would have expected a higher score
> with a test set inculded in the training corpus.
>
> makes sense ?
>
>
> Next steps :
> What path should I use to get better scores ? I read the 'optimize'
> section of the website which deals more with speed
> and of course I will appply all of this but I was interested in tips to
> get more quality if possible.
>
> look into domain adaptation if you have multiple training corpora,
> some of which is in-domain and some out-of-domain.
>
> Other than that, getting good bleu score is a research open question.
>
> Well done on getting this far
>
>
> Thanks
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
>

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

thanks will do.
Am I correct saying the tokenize / clean is mono thread ? or maybe I 
misconfigured something.


Le 06/08/2015 13:28, Barry Haddow a écrit :

Hi Vincent

It's a SIGKILL. Probably means it ran out of memory.

I'd recommend fast_align for this data set. Even if you manage to get 
it running with mgiza it will still take a week or so.


Just add
fast-align-settings = "-d -o -v"
to the TRAINING section of ems, and make sure that fast_align is in 
your external-bin-dir.


cheers - Barry

On 06/08/15 08:40, Vincent Nguyen wrote:


so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of: 
/home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look at 
the system descriptions to see what the authors did. In fact it's 
worth looking at the WMT14 descriptions (WMT15 will be available 
next month) to see how state-of-the-art systems are built.


For fr-en or en-fr, the first thing to look at is the data. There 
are some large data sets released for WMT and you can get a good 
gain from just crunching more data (monolingual and parallel). 
Unfortunately this takes more resources (disk, cpu etc) so you may 
run into trouble here.


The hierarchical models are much bigger so yes you will need more 
disk. For fr-en/en-fr it's probably not worth the extra effort,


cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my 
internal TMX


thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced 
opinions.


Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can only be
used in single-threaded decoding.
2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
KenLM)

true
3) Compact Mode is faster than onDisk with a short test (77 
segments 96

seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the 
evaluation (I

know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a higher 
score

with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the 'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested in 
tips to

get more quality if possible.

look into domain adaptation if you have multiple training corpora,
some of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open question.

Well done on getting this far


Thanks



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support










The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Barry Haddow]

Hi Vincent

It's a SIGKILL. Probably means it ran out of memory.

I'd recommend fast_align for this data set. Even if you manage to get it 
running with mgiza it will still take a week or so.


Just add
fast-align-settings = "-d -o -v"
to the TRAINING section of ems, and make sure that fast_align is in your 
external-bin-dir.


cheers - Barry

On 06/08/15 08:40, Vincent Nguyen wrote:


so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of: 
/home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look at 
the system descriptions to see what the authors did. In fact it's 
worth looking at the WMT14 descriptions (WMT15 will be available next 
month) to see how state-of-the-art systems are built.


For fr-en or en-fr, the first thing to look at is the data. There are 
some large data sets released for WMT and you can get a good gain 
from just crunching more data (monolingual and parallel). 
Unfortunately this takes more resources (disk, cpu etc) so you may 
run into trouble here.


The hierarchical models are much bigger so yes you will need more 
disk. For fr-en/en-fr it's probably not worth the extra effort,


cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my 
internal TMX


thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced 
opinions.


Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can only be
used in single-threaded decoding.
2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
KenLM)

true
3) Compact Mode is faster than onDisk with a short test (77 
segments 96

seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the 
evaluation (I

know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a higher 
score

with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the 'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested in 
tips to

get more quality if possible.

look into domain adaptation if you have multiple training corpora,
some of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open question.

Well done on getting this far


Thanks



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support








The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

so I dropped my hierarchical model since I got an error.
Switched back to the "more data" by adding the Giga FR EN source
but now another error pops un running Giza Inverse :

Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(2) running giza @ Wed Aug  5 21:03:56 CEST 2015
(2.1a) running snt2cooc fr-en @ Wed Aug  5 21:03:56 CEST 2015
Executing: mkdir -p /home/moses/working/training/giza-inverse.7
Executing: /home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

line 1000
line 2000

...
line 6609000
line 661
ERROR: Execution of: 
/home/moses/working/bin/training-tools/mgizapp/snt2cooc 
/home/moses/working/training/giza-inverse.7/fr-en.cooc 
/home/moses/working/training/prepared.7/en.vcb 
/home/moses/working/training/prepared.7/fr.vcb 
/home/moses/working/training/prepared.7/fr-en-int-train.snt

  died with signal 9, without coredump


any clue what signal 9 means ?



Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look at the 
system descriptions to see what the authors did. In fact it's worth 
looking at the WMT14 descriptions (WMT15 will be available next month) 
to see how state-of-the-art systems are built.


For fr-en or en-fr, the first thing to look at is the data. There are 
some large data sets released for WMT and you can get a good gain from 
just crunching more data (monolingual and parallel). Unfortunately 
this takes more resources (disk, cpu etc) so you may run into trouble 
here.


The hierarchical models are much bigger so yes you will need more 
disk. For fr-en/en-fr it's probably not worth the extra effort,


cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my 
internal TMX


thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can only be
used in single-threaded decoding.
2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
KenLM)

true
3) Compact Mode is faster than onDisk with a short test (77 
segments 96

seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a higher 
score

with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the 'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested in 
tips to

get more quality if possible.

look into domain adaptation if you have multiple training corpora,
some of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open question.

Well done on getting this far


Thanks



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support






___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Hieu Hoang]

quite right. If anyone wants to volunteer, the out-of-disk-space error
would most likely to be in the sorting.

For extract, the likeliest place is
   scripts/generic/extract-parallel.perl
wherever you see the RunFork() method being called

Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

On 5 August 2015 at 14:39, Kenneth Heafield  wrote:

> Looking for a perl volunteer to:
>
> 1. Always run commands under set -e -o pipefail conditions so errors are
> likely to be reported in the return code.
>
> 2. Actually check the return code and die on failure.
>
> It shouldn't be guesswork when one runs out of disk space.
>
> On 08/05/15 08:43, Hieu Hoang wrote:
> > are you sure it didn't run out of disk space again? check in the
> > TRAINING_extract.*.STDERR file for messages.
> >
> > Also, because extract and scoring it is run in parallel, the error
> > messages sometimes overwrite each other so you don't get clear messages.
> > you have to use your intuition
> >
> >
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Kenneth Heafield]

Looking for a perl volunteer to:

1. Always run commands under set -e -o pipefail conditions so errors are
likely to be reported in the return code. 

2. Actually check the return code and die on failure. 

It shouldn't be guesswork when one runs out of disk space. 

On 08/05/15 08:43, Hieu Hoang wrote:
> are you sure it didn't run out of disk space again? check in the 
> TRAINING_extract.*.STDERR file for messages.
>
> Also, because extract and scoring it is run in parallel, the error 
> messages sometimes overwrite each other so you don't get clear messages. 
> you have to use your intuition
>
>

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

I doubt for both.
extract job was finished at 1:42 AM, the other error popped much later.
the what() refers to consolidate-main.cpp not to the gzip stuff


Le 05/08/2015 10:18, Barry Haddow a écrit :
> Could it be this zlib bug?
>
> http://permalink.gmane.org/gmane.comp.nlp.moses.user/10151
>
> On 05/08/15 08:43, Hieu Hoang wrote:
>> are you sure it didn't run out of disk space again? check in the 
>> TRAINING_extract.*.STDERR file for messages.
>>
>> Also, because extract and scoring it is run in parallel, the error 
>> messages sometimes overwrite each other so you don't get clear 
>> messages. you have to use your intuition
>>
>> On 05/08/2015 11:31, Vincent Nguyen wrote:
>>>
>>> I increased the disk space but the hierarchical training crashes for 
>>> another reason I do not understand.
>>> it's at build_ttable time
>>> I attached the error + config
>>> cheers,
>>> Vincent
>>>
>>> Le 04/08/2015 17:28, Barry Haddow a écrit :
 Hi Vincent

 If you are comparing to the results of WMT11, then you can look at 
 the system descriptions to see what the authors did. In fact it's 
 worth looking at the WMT14 descriptions (WMT15 will be available 
 next month) to see how state-of-the-art systems are built.

 For fr-en or en-fr, the first thing to look at is the data. There 
 are some large data sets released for WMT and you can get a good 
 gain from just crunching more data (monolingual and parallel). 
 Unfortunately this takes more resources (disk, cpu etc) so you may 
 run into trouble here.

 The hierarchical models are much bigger so yes you will need more 
 disk. For fr-en/en-fr it's probably not worth the extra effort,

 cheers - Barry

 On 04/08/15 15:58, Vincent Nguyen wrote:
> thanks for your insights.
>
> I am just stuck by the Bleu difference between my 26 and the 30 of
> WMT11, and some results of WMT14 close to 36 or even 39
>
> I am currently having trouble with hierarchical rule set instead of
> lexical reordering
> wondering if I will get better results but I have an error message
> filesystem root low disk space before it crashes.
> is this model taking more disk space in some ways ?
>
> I will next try to use more corpora of which in domain with my 
> internal TMX
>
> thanks for your answers.
>
> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>>
>> On 03/08/2015 13:00, Vincent Nguyen wrote:
>>> Hi,
>>>
>>> Just a heads up on some EMS results, to get your experienced 
>>> opinions.
>>>
>>> Corpus: Europarlv7 + NC2010
>>> fr => en
>>> Evaluation NC2011.
>>>
>>> 1) IRSTLM vs KenLM is much slower for training / tuning.
>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>> used in single-threaded decoding.
>>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 
>>> with KenLM)
>> true
>>> 3) Compact Mode is faster than onDisk with a short test (77 
>>> segments 96
>>> seconds, vs 126 seconds)
>> true
>>> 4) One last thing I do not understand though :
>>> For sake of checking, I replaced NC2011 by NC2010 in the 
>>> evaluation (I
>>> know since NC2010 is part of training, should not be relevant)
>>> I got roughly the same BLEU score. I would have expected a 
>>> higher score
>>> with a test set inculded in the training corpus.
>>>
>>> makes sense ?
>>>
>>>
>>> Next steps :
>>> What path should I use to get better scores ? I read the 'optimize'
>>> section of the website which deals more with speed
>>> and of course I will appply all of this but I was interested in 
>>> tips to
>>> get more quality if possible.
>> look into domain adaptation if you have multiple training corpora,
>> some of which is in-domain and some out-of-domain.
>>
>> Other than that, getting good bleu score is a research open 
>> question.
>>
>> Well done on getting this far
>>>
>>> Thanks
>>>
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


>>>
>>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Barry Haddow]

Could it be this zlib bug?

http://permalink.gmane.org/gmane.comp.nlp.moses.user/10151

On 05/08/15 08:43, Hieu Hoang wrote:
> are you sure it didn't run out of disk space again? check in the 
> TRAINING_extract.*.STDERR file for messages.
>
> Also, because extract and scoring it is run in parallel, the error 
> messages sometimes overwrite each other so you don't get clear 
> messages. you have to use your intuition
>
> On 05/08/2015 11:31, Vincent Nguyen wrote:
>>
>> I increased the disk space but the hierarchical training crashes for 
>> another reason I do not understand.
>> it's at build_ttable time
>> I attached the error + config
>> cheers,
>> Vincent
>>
>> Le 04/08/2015 17:28, Barry Haddow a écrit :
>>> Hi Vincent
>>>
>>> If you are comparing to the results of WMT11, then you can look at 
>>> the system descriptions to see what the authors did. In fact it's 
>>> worth looking at the WMT14 descriptions (WMT15 will be available 
>>> next month) to see how state-of-the-art systems are built.
>>>
>>> For fr-en or en-fr, the first thing to look at is the data. There 
>>> are some large data sets released for WMT and you can get a good 
>>> gain from just crunching more data (monolingual and parallel). 
>>> Unfortunately this takes more resources (disk, cpu etc) so you may 
>>> run into trouble here.
>>>
>>> The hierarchical models are much bigger so yes you will need more 
>>> disk. For fr-en/en-fr it's probably not worth the extra effort,
>>>
>>> cheers - Barry
>>>
>>> On 04/08/15 15:58, Vincent Nguyen wrote:
 thanks for your insights.

 I am just stuck by the Bleu difference between my 26 and the 30 of
 WMT11, and some results of WMT14 close to 36 or even 39

 I am currently having trouble with hierarchical rule set instead of
 lexical reordering
 wondering if I will get better results but I have an error message
 filesystem root low disk space before it crashes.
 is this model taking more disk space in some ways ?

 I will next try to use more corpora of which in domain with my 
 internal TMX

 thanks for your answers.

 Le 04/08/2015 16:02, Hieu Hoang a écrit :
>
> On 03/08/2015 13:00, Vincent Nguyen wrote:
>> Hi,
>>
>> Just a heads up on some EMS results, to get your experienced 
>> opinions.
>>
>> Corpus: Europarlv7 + NC2010
>> fr => en
>> Evaluation NC2011.
>>
>> 1) IRSTLM vs KenLM is much slower for training / tuning.
> that sounds right. KenLM is also multithreaded, IRSTLM can only be
> used in single-threaded decoding.
>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
>> KenLM)
> true
>> 3) Compact Mode is faster than onDisk with a short test (77 
>> segments 96
>> seconds, vs 126 seconds)
> true
>> 4) One last thing I do not understand though :
>> For sake of checking, I replaced NC2011 by NC2010 in the 
>> evaluation (I
>> know since NC2010 is part of training, should not be relevant)
>> I got roughly the same BLEU score. I would have expected a higher 
>> score
>> with a test set inculded in the training corpus.
>>
>> makes sense ?
>>
>>
>> Next steps :
>> What path should I use to get better scores ? I read the 'optimize'
>> section of the website which deals more with speed
>> and of course I will appply all of this but I was interested in 
>> tips to
>> get more quality if possible.
> look into domain adaptation if you have multiple training corpora,
> some of which is in-domain and some out-of-domain.
>
> Other than that, getting good bleu score is a research open question.
>
> Well done on getting this far
>>
>> Thanks
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

>>>
>>>
>>
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Hieu Hoang]

are you sure it didn't run out of disk space again? check in the 
TRAINING_extract.*.STDERR file for messages.

Also, because extract and scoring it is run in parallel, the error 
messages sometimes overwrite each other so you don't get clear messages. 
you have to use your intuition

On 05/08/2015 11:31, Vincent Nguyen wrote:
>
> I increased the disk space but the hierarchical training crashes for 
> another reason I do not understand.
> it's at build_ttable time
> I attached the error + config
> cheers,
> Vincent
>
> Le 04/08/2015 17:28, Barry Haddow a écrit :
>> Hi Vincent
>>
>> If you are comparing to the results of WMT11, then you can look at 
>> the system descriptions to see what the authors did. In fact it's 
>> worth looking at the WMT14 descriptions (WMT15 will be available next 
>> month) to see how state-of-the-art systems are built.
>>
>> For fr-en or en-fr, the first thing to look at is the data. There are 
>> some large data sets released for WMT and you can get a good gain 
>> from just crunching more data (monolingual and parallel). 
>> Unfortunately this takes more resources (disk, cpu etc) so you may 
>> run into trouble here.
>>
>> The hierarchical models are much bigger so yes you will need more 
>> disk. For fr-en/en-fr it's probably not worth the extra effort,
>>
>> cheers - Barry
>>
>> On 04/08/15 15:58, Vincent Nguyen wrote:
>>> thanks for your insights.
>>>
>>> I am just stuck by the Bleu difference between my 26 and the 30 of
>>> WMT11, and some results of WMT14 close to 36 or even 39
>>>
>>> I am currently having trouble with hierarchical rule set instead of
>>> lexical reordering
>>> wondering if I will get better results but I have an error message
>>> filesystem root low disk space before it crashes.
>>> is this model taking more disk space in some ways ?
>>>
>>> I will next try to use more corpora of which in domain with my 
>>> internal TMX
>>>
>>> thanks for your answers.
>>>
>>> Le 04/08/2015 16:02, Hieu Hoang a écrit :

 On 03/08/2015 13:00, Vincent Nguyen wrote:
> Hi,
>
> Just a heads up on some EMS results, to get your experienced 
> opinions.
>
> Corpus: Europarlv7 + NC2010
> fr => en
> Evaluation NC2011.
>
> 1) IRSTLM vs KenLM is much slower for training / tuning.
 that sounds right. KenLM is also multithreaded, IRSTLM can only be
 used in single-threaded decoding.
> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
> KenLM)
 true
> 3) Compact Mode is faster than onDisk with a short test (77 
> segments 96
> seconds, vs 126 seconds)
 true
> 4) One last thing I do not understand though :
> For sake of checking, I replaced NC2011 by NC2010 in the 
> evaluation (I
> know since NC2010 is part of training, should not be relevant)
> I got roughly the same BLEU score. I would have expected a higher 
> score
> with a test set inculded in the training corpus.
>
> makes sense ?
>
>
> Next steps :
> What path should I use to get better scores ? I read the 'optimize'
> section of the website which deals more with speed
> and of course I will appply all of this but I was interested in 
> tips to
> get more quality if possible.
 look into domain adaptation if you have multiple training corpora,
 some of which is in-domain and some out-of-domain.

 Other than that, getting good bleu score is a research open question.

 Well done on getting this far
>
> Thanks
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>

-- 
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

I increased the disk space but the hierarchical training crashes for 
another reason I do not understand.

it's at build_ttable time
I attached the error + config
cheers,
Vincent

Le 04/08/2015 17:28, Barry Haddow a écrit :

Hi Vincent

If you are comparing to the results of WMT11, then you can look at the 
system descriptions to see what the authors did. In fact it's worth 
looking at the WMT14 descriptions (WMT15 will be available next month) 
to see how state-of-the-art systems are built.


For fr-en or en-fr, the first thing to look at is the data. There are 
some large data sets released for WMT and you can get a good gain from 
just crunching more data (monolingual and parallel). Unfortunately 
this takes more resources (disk, cpu etc) so you may run into trouble 
here.


The hierarchical models are much bigger so yes you will need more 
disk. For fr-en/en-fr it's probably not worth the extra effort,


cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set instead of
lexical reordering
wondering if I will get better results but I have an error message
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my 
internal TMX


thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :


On 03/08/2015 13:00, Vincent Nguyen wrote:

Hi,

Just a heads up on some EMS results, to get your experienced opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

that sounds right. KenLM is also multithreaded, IRSTLM can only be
used in single-threaded decoding.
2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with 
KenLM)

true
3) Compact Mode is faster than onDisk with a short test (77 
segments 96

seconds, vs 126 seconds)

true

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a higher 
score

with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the 'optimize'
section of the website which deals more with speed
and of course I will appply all of this but I was interested in 
tips to

get more quality if possible.

look into domain adaptation if you have multiple training corpora,
some of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open question.

Well done on getting this far


Thanks



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support







### CONFIGURATION FILE FOR AN SMT EXPERIMENT ###


[GENERAL]

### directory in which experiment is run
#
working-dir = /home/moses/working

# specification of the language pair
input-extension = fr
output-extension = en
pair-extension = fr-en

### directories that contain tools and data
# 
# moses
moses-src-dir = /home/moses/mosesdecoder
#
# moses binaries
moses-bin-dir = $moses-src-dir/bin
#
# moses scripts
moses-script-dir = $moses-src-dir/scripts
#
# directory where GIZA++/MGIZA programs resides
external-bin-dir = /home/moses/working/bin/training-tools/mgizapp
#
# srilm
srilm-dir = $moses-src-dir/srilm/bin/i686-m64
#
# irstlm
irstlm-dir = $moses-src-dir/irstlm/bin
#
# randlm
#randlm-dir = $moses-src-dir/randlm/bin
#
# data
wmt12-data = $working-dir/data

### basic tools
#
# moses decoder
decoder = $moses-bin-dir/moses

# conversion of rule table into binary on-disk format
#ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
ttable-binarizer = "$moses-bin-dir/processPhraseTableMin"


# tokenizers - comment out if all your data is already tokenized
input-tokenizer = "$moses-script-dir/tokenizer/tokenizer.perl -a -l 
$input-extension"
output-tokenizer = "$moses-script-dir/tokenizer/tokenizer.perl -a -l 
$output-extension"

# truecasers - comment out if you do not use the truecaser
input-truecaser = $moses-script-dir/recaser/truecase.perl
output-truecaser = $moses-script-dir/recaser/truecase.perl
detruecaser = $moses-script-dir/recaser/detruecase.perl

# lowercaser - comment out if you use truecasing
#input-lowercaser = $moses-script-dir/tokenizer/lowercase.perl
#output-lowercaser = $moses-script-dir/tokenizer/lowercase.perl

### generic parallelizer for cluster and multi-core machines
# you may speci

Re: [Moses-support] EMS results - makes sense ?

[Barry Haddow]

Hi Vincent

If you are comparing to the results of WMT11, then you can look at the 
system descriptions to see what the authors did. In fact it's worth 
looking at the WMT14 descriptions (WMT15 will be available next month) 
to see how state-of-the-art systems are built.

For fr-en or en-fr, the first thing to look at is the data. There are 
some large data sets released for WMT and you can get a good gain from 
just crunching more data (monolingual and parallel). Unfortunately this 
takes more resources (disk, cpu etc) so you may run into trouble here.

The hierarchical models are much bigger so yes you will need more disk. 
For fr-en/en-fr it's probably not worth the extra effort,

cheers - Barry

On 04/08/15 15:58, Vincent Nguyen wrote:
> thanks for your insights.
>
> I am just stuck by the Bleu difference between my 26 and the 30 of
> WMT11, and some results of WMT14 close to 36 or even 39
>
> I am currently having trouble with hierarchical rule set instead of
> lexical reordering
> wondering if I will get better results but I have an error message
> filesystem root low disk space before it crashes.
> is this model taking more disk space in some ways ?
>
> I will next try to use more corpora of which in domain with my internal TMX
>
> thanks for your answers.
>
> Le 04/08/2015 16:02, Hieu Hoang a écrit :
>>
>> On 03/08/2015 13:00, Vincent Nguyen wrote:
>>> Hi,
>>>
>>> Just a heads up on some EMS results, to get your experienced opinions.
>>>
>>> Corpus: Europarlv7 + NC2010
>>> fr => en
>>> Evaluation NC2011.
>>>
>>> 1) IRSTLM vs KenLM is much slower for training / tuning.
>> that sounds right. KenLM is also multithreaded, IRSTLM can only be
>> used in single-threaded decoding.
>>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)
>> true
>>> 3) Compact Mode is faster than onDisk with a short test (77 segments 96
>>> seconds, vs 126 seconds)
>> true
>>> 4) One last thing I do not understand though :
>>> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
>>> know since NC2010 is part of training, should not be relevant)
>>> I got roughly the same BLEU score. I would have expected a higher score
>>> with a test set inculded in the training corpus.
>>>
>>> makes sense ?
>>>
>>>
>>> Next steps :
>>> What path should I use to get better scores ? I read the 'optimize'
>>> section of the website which deals more with speed
>>> and of course I will appply all of this but I was interested in tips to
>>> get more quality if possible.
>> look into domain adaptation if you have multiple training corpora,
>> some of which is in-domain and some out-of-domain.
>>
>> Other than that, getting good bleu score is a research open question.
>>
>> Well done on getting this far
>>>
>>> Thanks
>>>
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

thanks for your insights.

I am just stuck by the Bleu difference between my 26 and the 30 of 
WMT11, and some results of WMT14 close to 36 or even 39

I am currently having trouble with hierarchical rule set instead of 
lexical reordering
wondering if I will get better results but I have an error message 
filesystem root low disk space before it crashes.
is this model taking more disk space in some ways ?

I will next try to use more corpora of which in domain with my internal TMX

thanks for your answers.

Le 04/08/2015 16:02, Hieu Hoang a écrit :
>
>
> On 03/08/2015 13:00, Vincent Nguyen wrote:
>> Hi,
>>
>> Just a heads up on some EMS results, to get your experienced opinions.
>>
>> Corpus: Europarlv7 + NC2010
>> fr => en
>> Evaluation NC2011.
>>
>> 1) IRSTLM vs KenLM is much slower for training / tuning.
> that sounds right. KenLM is also multithreaded, IRSTLM can only be 
> used in single-threaded decoding.
>>
>> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)
> true
>>
>> 3) Compact Mode is faster than onDisk with a short test (77 segments 96
>> seconds, vs 126 seconds)
> true
>>
>> 4) One last thing I do not understand though :
>> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
>> know since NC2010 is part of training, should not be relevant)
>> I got roughly the same BLEU score. I would have expected a higher score
>> with a test set inculded in the training corpus.
>>
>> makes sense ?
>>
>>
>> Next steps :
>> What path should I use to get better scores ? I read the 'optimize'
>> section of the website which deals more with speed
>> and of course I will appply all of this but I was interested in tips to
>> get more quality if possible.
> look into domain adaptation if you have multiple training corpora, 
> some of which is in-domain and some out-of-domain.
>
> Other than that, getting good bleu score is a research open question.
>
> Well done on getting this far
>>
>>
>> Thanks
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS results - makes sense ?

[Hieu Hoang]

On 03/08/2015 13:00, Vincent Nguyen wrote:
> Hi,
>
> Just a heads up on some EMS results, to get your experienced opinions.
>
> Corpus: Europarlv7 + NC2010
> fr => en
> Evaluation NC2011.
>
> 1) IRSTLM vs KenLM is much slower for training / tuning.
that sounds right. KenLM is also multithreaded, IRSTLM can only be used 
in single-threaded decoding.
>
> 2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)
true
>
> 3) Compact Mode is faster than onDisk with a short test (77 segments 96
> seconds, vs 126 seconds)
true
>
> 4) One last thing I do not understand though :
> For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I
> know since NC2010 is part of training, should not be relevant)
> I got roughly the same BLEU score. I would have expected a higher score
> with a test set inculded in the training corpus.
>
> makes sense ?
>
>
> Next steps :
> What path should I use to get better scores ? I read the 'optimize'
> section of the website which deals more with speed
> and of course I will appply all of this but I was interested in tips to
> get more quality if possible.
look into domain adaptation if you have multiple training corpora, some 
of which is in-domain and some out-of-domain.

Other than that, getting good bleu score is a research open question.

Well done on getting this far
>
>
> Thanks
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

-- 
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] EMS results - makes sense ?

[Vincent Nguyen]

Hi,

Just a heads up on some EMS results, to get your experienced opinions.

Corpus: Europarlv7 + NC2010
fr => en
Evaluation NC2011.

1) IRSTLM vs KenLM is much slower for training / tuning.

2) BLEU results are almost the same (25.7 with Irstlm, 26.14 with KenLM)

3) Compact Mode is faster than onDisk with a short test (77 segments 96 
seconds, vs 126 seconds)

4) One last thing I do not understand though :
For sake of checking, I replaced NC2011 by NC2010 in the evaluation (I 
know since NC2010 is part of training, should not be relevant)
I got roughly the same BLEU score. I would have expected a higher score 
with a test set inculded in the training corpus.

makes sense ?


Next steps :
What path should I use to get better scores ? I read the 'optimize' 
section of the website which deals more with speed
and of course I will appply all of this but I was interested in tips to 
get more quality if possible.


Thanks



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support