Re: [Moses-support] EMS system error

[Philipp Koehn]

Hi,

as long as you get error messages from experiment.perl, nothing will be
executed.

The error message indicates that IRSTLM is not installed. You do not need
it (you also do not need SRILM), so you should just comment out references
to them and use KENLM.

If everything runs through, the BLEU score will be in the file
evaluation/report.1

-phi

On Mon, Dec 28, 2015 at 7:04 AM, Sunayana Gawde 
wrote:

> Hi all,
>
> I tried giving my training tuning and test data in respective files and
> then i did the change in config file as: home-dir = /home/sunayana
>
> then i used the command :
>
> nohup nice ~/mosesdecoder/scripts/ems/experiment.perl -config config -exec &> 
> log &
>
> it gives me this :
> sunayana@sunayana:~/working/experiments$ nohup nice 
> ~/mosesdecoder/scripts/ems/experiment.perl -config config -exec &> log &
> [1] 5823
>
>
> where do i check for BLEU score?
>
> I checked the log file here :/home/sunayana/working/experiments/steps
>
> there i get:
>
> nohup: ignoring input
> STARTING UP AS PROCESS 5823 ON sunayana AT Mon Dec 28 17:27:04 IST 2015
> LOAD CONFIG...
> find: `/home/sunayana/irstlm/bin*': No such file or directory
> GENERAL:irstlm-dir: file /home/sunayana/irstlm/bin does not exist!
> Died at /home/sunayana/mosesdecoder/scripts/ems/experiment.perl line 437.
>
> I also installed image magick, ghost view tool and Graphviz
>
> Please help me in this.
>
>
>
> --
> *Thanks & Regards*
>
> Ms. Sunayana R. Gawde.
>
> DCST, Goa University.
> * P**leas**e don't print t**his e-mail unles**s you really need to.*
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] How much tuning data?

[Philipp Koehn]

Hi,

the size of the tuning set is one factor in the variance between different
tuning runs.

Run MERT (or kbmira) multiple times, to see what test BLEU scores you will
get. If they vary too much between tuning runs, then a larger tuning set
would be better.

The motivation to keep variance low is that you can actually observe the
impact of your improved system over the baseline.

-phi

On Mon, Dec 28, 2015 at 11:16 AM, Vincent Nguyen  wrote:

>
> this is fine for tuning. if you want to make it quicker, drag it down to
> 1000 sentences.
>
>
> Le 28/12/2015 16:37, Read, James C a écrit :
>
> Hi,
>
>
> I'm setting up some Moses baseline systems for various language pairs to
> compare the systems against my own work. I've largely been following the
> baseline tutorial here http://www.statmt.org/moses/?n=moses.baseline
>
>
> What would a fair amount of tuning data be to do the Moses system justice?
> I've currently managed to isolate a set of about 1,900 sentences which are
> roughly common to all language pairs. Would that be enough? To much? Not
> enough?
>
>
> James
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Segmentation Fault (Core Dumped)

[Rajen Chatterjee]

Hi Gozde,

Apart from the possible problem of memory, another problem can be that your
mapping do not generate the LM factors.
I think that for "t2" instead of 0-0 should have 0-0,1,2 (since you are
using 3 LM, one for each factor), and similar changes will be required for
another decoding path.

On Mon, Dec 28, 2015 at 1:30 PM, Hieu Hoang  wrote:

> you should start with simple factored models 1st, perhaps using only 1
> translation model with 2 factors. Then move onto 1 translation model and 1
> generation model.
>
> The factored you are difficult to control, they use a lot of memory and
> takes a lot of time. You may be getting errors because it runs out of
> memory.
>
>
> On 28/12/15 10:01, gozde gul wrote:
>
> Hi,
>
> I am trying to perform a 3 factored translation from English to Turkish.
> My example input is as follows:
>
> En: Life+NNP|Life|NNP end+VBZ_never+RB|end|VBZ_never+RB but+CC|but|CC
> earthly+JJ|earthly|JJ life+NN|life|NN do+VBZ|do|VBZ .+.|.|.
> Tr:  Hayat|hayat|+Noun+A3sg+Pnon+Nom hiç|hiç|+Adverb
> bitmez|bit|+Verb+Neg+Aor+A3sg fakat|fakat|+Conj
> dünyadaki|dünya|+Noun+A3sg+Pnon+Loc^DB+Adj+Rel
> hayat|hayat|+Noun+A3sg+Pnon+Nom biter|bit|+Verb+Pos+Aor+A3sg .|.|+Punc
>
> My translation and generation factors and decoding steps are as follows. I
> am pretty sure they are correct:
> --translation-factors 1-1+2-2+0-0 \
> --generation-factors 1,2-0 \
> --decoding-steps t2:t0,t1,g0
>
> I have created language models for all 3 factors with irstlm with the
> steps explained in moses website.
>
> If I train with the following model, it creates a moses.ini. When I
> manually check the phrase tables and generation table they look meaningful.
> ~/mosesdecoder/scripts/training/train-model.perl \
> --parallel --mgiza \
> --external-bin-dir ~/workspace/bin/training-tools/mgizapp \
> --root-dir ~/FactoredModel/SmallModel/  \
> --corpus
> ~/FactoredModel/SmallModel/factored-corpus/training/korpus_1000K.en-tr.KO.recleaned_new
> \
> --f en --e tr --alignment grow-diag-final-and \
> --reordering msd-bidirectional-fe \
> --lm 0:3:$HOME/corpus/FilteredCorpus/training/lm/surLM/sur.lm.blm.tr:8 \
> --lm 1:3:$HOME/corpus/FilteredCorpus/training/lm/lemmaLM/lemma.lm.blm.tr:8
>  \
> --lm 2:3:$HOME/corpus/FilteredCorpus/training/lm/postagLM/
> postags.lm.blm.tr:8 \
> --alignment-factors 1-1 \
> --translation-factors 1-1+2-2+0-0 \
> --generation-factors 1,2-0 \
> --decoding-steps t2:t0,t1,g0 >& ~/FactoredModel/trainingSmall3lm.out
>
> But when I try to decode a very simple one-line sentence, I get a
> "Segmentation fault (Core Dumped)" message. You can see the detailed
> decoding log here
> . I tried many
> things and I'm in a dead end, so I would really appreciate your help.
>
> Thanks,
>
> Gozde
>
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
-Regards,
 Rajen Chatterjee.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] How much tuning data?

[Vincent Nguyen]

this is fine for tuning. if you want to make it quicker, drag it down to 
1000 sentences.


Le 28/12/2015 16:37, Read, James C a écrit :


Hi,


I'm setting up some Moses baseline systems for various language pairs 
to compare the systems against my own work. I've largely been 
following the baseline tutorial here 
http://www.statmt.org/moses/?n=moses.baseline



What would a fair amount of tuning data be to do the Moses system 
justice? I've currently managed to isolate a set of about 1,900 
sentences which are roughly common to all language pairs. Would that 
be enough? To much? Not enough?



James



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] How much tuning data?

[Read, James C]

Hi,


I'm setting up some Moses baseline systems for various language pairs to 
compare the systems against my own work. I've largely been following the 
baseline tutorial here http://www.statmt.org/moses/?n=moses.baseline


What would a fair amount of tuning data be to do the Moses system justice? I've 
currently managed to isolate a set of about 1,900 sentences which are roughly 
common to all language pairs. Would that be enough? To much? Not enough?


James
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Segmentation Fault (Core Dumped)

[Hieu Hoang]

you should start with simple factored models 1st, perhaps using only 1 
translation model with 2 factors. Then move onto 1 translation model and 
1 generation model.


The factored you are difficult to control, they use a lot of memory and 
takes a lot of time. You may be getting errors because it runs out of 
memory.


On 28/12/15 10:01, gozde gul wrote:

Hi,

I am trying to perform a 3 factored translation from English to 
Turkish. My example input is as follows:


En: Life+NNP|Life|NNP end+VBZ_never+RB|end|VBZ_never+RB but+CC|but|CC 
earthly+JJ|earthly|JJ life+NN|life|NN do+VBZ|do|VBZ .+.|.|.
Tr:  Hayat|hayat|+Noun+A3sg+Pnon+Nom hiç|hiç|+Adverb 
bitmez|bit|+Verb+Neg+Aor+A3sg fakat|fakat|+Conj 
dünyadaki|dünya|+Noun+A3sg+Pnon+Loc^DB+Adj+Rel 
hayat|hayat|+Noun+A3sg+Pnon+Nom biter|bit|+Verb+Pos+Aor+A3sg .|.|+Punc


My translation and generation factors and decoding steps are as 
follows. I am pretty sure they are correct:

--translation-factors 1-1+2-2+0-0 \
--generation-factors 1,2-0 \
--decoding-steps t2:t0,t1,g0

I have created language models for all 3 factors with irstlm with the 
steps explained in moses website.


If I train with the following model, it creates a moses.ini. When I 
manually check the phrase tables and generation table they look 
meaningful.

~/mosesdecoder/scripts/training/train-model.perl \
--parallel --mgiza \
--external-bin-dir ~/workspace/bin/training-tools/mgizapp \
--root-dir ~/FactoredModel/SmallModel/  \
--corpus 
~/FactoredModel/SmallModel/factored-corpus/training/korpus_1000K.en-tr.KO.recleaned_new 
\

--f en --e tr --alignment grow-diag-final-and \
--reordering msd-bidirectional-fe \
--lm 0:3:$HOME/corpus/FilteredCorpus/training/lm/surLM/sur.lm.blm.tr:8 
 \
--lm 
1:3:$HOME/corpus/FilteredCorpus/training/lm/lemmaLM/lemma.lm.blm.tr:8 
 \
--lm 
2:3:$HOME/corpus/FilteredCorpus/training/lm/postagLM/postags.lm.blm.tr:8 
 \

--alignment-factors 1-1 \
--translation-factors 1-1+2-2+0-0 \
--generation-factors 1,2-0 \
--decoding-steps t2:t0,t1,g0 >& ~/FactoredModel/trainingSmall3lm.out

But when I try to decode a very simple one-line sentence, I get a 
"Segmentation fault (Core Dumped)" message. You can see the detailed 
decoding log here 
. I tried 
many things and I'm in a dead end, so I would really appreciate your help.


Thanks,

Gozde



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] EMS system error

[Sunayana Gawde]

Hi all,

I tried giving my training tuning and test data in respective files and
then i did the change in config file as: home-dir = /home/sunayana

then i used the command :

nohup nice ~/mosesdecoder/scripts/ems/experiment.perl -config config
-exec &> log &

it gives me this :
sunayana@sunayana:~/working/experiments$ nohup nice
~/mosesdecoder/scripts/ems/experiment.perl -config config -exec &> log
&
[1] 5823


where do i check for BLEU score?

I checked the log file here :/home/sunayana/working/experiments/steps

there i get:

nohup: ignoring input
STARTING UP AS PROCESS 5823 ON sunayana AT Mon Dec 28 17:27:04 IST 2015
LOAD CONFIG...
find: `/home/sunayana/irstlm/bin*': No such file or directory
GENERAL:irstlm-dir: file /home/sunayana/irstlm/bin does not exist!
Died at /home/sunayana/mosesdecoder/scripts/ems/experiment.perl line 437.

I also installed image magick, ghost view tool and Graphviz

Please help me in this.



-- 
*Thanks & Regards*

Ms. Sunayana R. Gawde.

DCST, Goa University.
* P**leas**e don't print t**his e-mail unles**s you really need to.*
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] moses build error

[Hieu Hoang]

unless you have a specific reason to use SRILM or IRSTLM, stick with KenLM.

This is built into Moses, you're less likely to get compile errors

On 28/12/15 11:35, Adel Khalifa wrote:
...failed gcc.link 
mert/bin/gcc-5.2.1/release/link-static/threading-multi/vocabulary_test...
...skipped 
vocabulary_test.passed 
for lack of 
vocabulary_test...

...failed updating 38 targets...
...skipped 116 targets...
...updated 858 targets...
The build failed.  If you need support, run:
  /usr/bin/bjam --with-srilm=/SMT/srilm --with-irstlm=/SMT/irstlm 
--with-giza=/SMT/bin -j2 --debug-configuration -d2 |gzip >build.log.gz

then attach build.log.gz to your e-mail.
You MUST do 3 things before sending to the mailing list:
   1. Subscribe to the mailing list at 
http://mailman.mit.edu/mailman/listinfo/moses-support

   2. Attach build.log.gz to your e-mail
   3. Say what is the EXACT command you executed when you got the error
ERROR
root@ubuntuSrv:~/SMT/moses# /usr/bin/bjam --with-srilm=/SMT/srilm 
--with-irstlm=/SMT/irstlm --with-giza=/SMT/bin -j2 
--debug-configuration -d2 |gzip >build.log.gz

sh: 1: /SMT/srilm/sbin/machine-type: not found



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Using multiple translation model (Paper)

[Hieu Hoang]

there are many but i guess a good paper would be
   http://www.isca-speech.org/archive/iwslt_11/papers/sltb_136.pdf

On 26/12/15 10:34, Cyrine NASRI wrote:

Hello,

I would like to know if there is a paper which describe how Moses do 
to select the best translation when We use 2 differents translation 
tables.


Best regards





___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] moses build error

[Adel Khalifa]

...failed gcc.link
mert/bin/gcc-5.2.1/release/link-static/threading-multi/vocabulary_test...
...skipped
vocabulary_test.passed
for lack of
vocabulary_test...
...failed updating 38 targets...
...skipped 116 targets...
...updated 858 targets...
The build failed.  If you need support, run:
  /usr/bin/bjam --with-srilm=/SMT/srilm --with-irstlm=/SMT/irstlm
--with-giza=/SMT/bin -j2 --debug-configuration -d2 |gzip >build.log.gz
then attach build.log.gz to your e-mail.
You MUST do 3 things before sending to the mailing list:
   1. Subscribe to the mailing list at
http://mailman.mit.edu/mailman/listinfo/moses-support
   2. Attach build.log.gz to your e-mail
   3. Say what is the EXACT command you executed when you got the error
ERROR
root@ubuntuSrv:~/SMT/moses# /usr/bin/bjam --with-srilm=/SMT/srilm
--with-irstlm=/SMT/irstlm --with-giza=/SMT/bin -j2 --debug-configuration
-d2 |gzip >build.log.gz
sh: 1: /SMT/srilm/sbin/machine-type: not found


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Segmentation Fault (Core Dumped)

[gozde gul]

Hi,

I am trying to perform a 3 factored translation from English to Turkish. My
example input is as follows:

En: Life+NNP|Life|NNP end+VBZ_never+RB|end|VBZ_never+RB but+CC|but|CC
earthly+JJ|earthly|JJ life+NN|life|NN do+VBZ|do|VBZ .+.|.|.
Tr:  Hayat|hayat|+Noun+A3sg+Pnon+Nom hiç|hiç|+Adverb
bitmez|bit|+Verb+Neg+Aor+A3sg fakat|fakat|+Conj
dünyadaki|dünya|+Noun+A3sg+Pnon+Loc^DB+Adj+Rel
hayat|hayat|+Noun+A3sg+Pnon+Nom biter|bit|+Verb+Pos+Aor+A3sg .|.|+Punc

My translation and generation factors and decoding steps are as follows. I
am pretty sure they are correct:
--translation-factors 1-1+2-2+0-0 \
--generation-factors 1,2-0 \
--decoding-steps t2:t0,t1,g0

I have created language models for all 3 factors with irstlm with the steps
explained in moses website.

If I train with the following model, it creates a moses.ini. When I
manually check the phrase tables and generation table they look meaningful.
~/mosesdecoder/scripts/training/train-model.perl \
--parallel --mgiza \
--external-bin-dir ~/workspace/bin/training-tools/mgizapp \
--root-dir ~/FactoredModel/SmallModel/  \
--corpus
~/FactoredModel/SmallModel/factored-corpus/training/korpus_1000K.en-tr.KO.recleaned_new
\
--f en --e tr --alignment grow-diag-final-and \
--reordering msd-bidirectional-fe \
--lm 0:3:$HOME/corpus/FilteredCorpus/training/lm/surLM/sur.lm.blm.tr:8 \
--lm 1:3:$HOME/corpus/FilteredCorpus/training/lm/lemmaLM/lemma.lm.blm.tr:8 \
--lm 2:3:$HOME/corpus/FilteredCorpus/training/lm/postagLM/
postags.lm.blm.tr:8 \
--alignment-factors 1-1 \
--translation-factors 1-1+2-2+0-0 \
--generation-factors 1,2-0 \
--decoding-steps t2:t0,t1,g0 >& ~/FactoredModel/trainingSmall3lm.out

But when I try to decode a very simple one-line sentence, I get a
"Segmentation fault (Core Dumped)" message. You can see the detailed
decoding log here .
I tried many things and I'm in a dead end, so I would really appreciate
your help.

Thanks,

Gozde
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support