Re: [Moses-support] ERROR: Lexical reordering scoring failed

[Philipp Koehn]

Hi,

can you check if the line

inhaled corticosteroids ||| inhalative Kortikoichaemic ||| ischämische |||
mono other

really occurs in the extract.o file?

It should only have source / target / reordering status, not 4 entries.

Did something go wrong when the extract file was created (out of disk
space?).

Do you have "|||" in the corpus?

Did you try re-running extract & build reorder?

-phi


On Tue, Mar 28, 2017 at 10:03 AM, Anna Kupś  wrote:

>
> Hi,
>
> I got the following error while using Experiment Management System:
>
> ==> TRAINING_build-reordering.3 <==
>
> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
> sbin:/bin:/snap/bin:/snap/bin"
> cd /root/working/experiments
> echo 'starting at '`date`' on '`hostname`
> mkdir -p /root/working/experiments/training
>
> /root/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8
> -dont-zip -first-step 7 -last-step 7 -external-bin-dir
> /root/mosesdecoder/tools -f en -e de -alignment grow-diag-final-and
> -reordering msd-bidirectional-fe -extract-file 
> /root/working/experiments/model/extract.2
> -reordering-table /root/working/experiments/model/reordering-table.3
>
> echo 'finished at '`date`
> touch /root/working/experiments/steps/3/TRAINING_build-reordering.3.DONE
>
> ==> TRAINING_build-reordering.3.INFO <==
> lexicalized-reordering = msd-bidirectional-fe
> INPUT = USED /root/working/experiments/model/extract.2
> # reuse run 2 for TRAINING:extract-phrases
>
> ==> TRAINING_build-reordering.3.STDERR <==
> Using SCRIPTS_ROOTDIR: /root/mosesdecoder/scripts
> using gzip
> (7) learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
> (7.1) [no factors] learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
> (7.2) building tables @ Tue Mar 28 12:19:55 CEST 2017
> Executing: /root/mosesdecoder/scripts/../bin/lexical-reordering-score
> /root/working/experiments/model/extract.2.o.sorted.gz 0.5
> /root/working/experiments/model/reordering-table.3. --model "wbe msd
> wbe-msd-bidirectional-fe"
> Lexical Reordering Scorer
> scores lexical reordering models of several types (hierarchical,
> phrase-based and word-based-extraction
>
> ==> TRAINING_build-reordering.3.STDOUT <==
> starting at Di 28. Mär 12:19:54 CEST 2017 on machine-VirtualBox
>
> ==> TRAINING_build-reordering.3.STDERR <==
> terminate called after throwing an instance of 'FileFormatException'
>   what():  phrase-extract/lexical-reordering/score.cpp:260 in void
> split_line(const StringPiece&, StringPiece&, StringPiece&, StringPiece&,
> StringPiece&, StringPiece&, float&) threw FileFormatException because
> `errIndex == next.data()'.
> Invalid extract file format: inhaled corticosteroids ||| inhalative
> Kortikoichaemic ||| ischämische ||| mono other
> Aborted (core dumped)
> Exit code: 134
> *ERROR: Lexical reordering scoring failed at
> /root/mosesdecoder/scripts/training/train-model.perl line 1924.*
>
> Does someone know how to get rid of that problem?
>
> Thanks in advance and best regards,
> Anna
>
> --
> Anna Kupś
> +48 782 823 850 <+48%20782%20823%20850> | www.linkedin.com/in/annakups
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Lexical reordering scoring failed

[Anna Kupś]

Hi,

I got the following error while using Experiment Management System:

==> TRAINING_build-reordering.3 <==

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/snap/bin"
cd /root/working/experiments
echo 'starting at '`date`' on '`hostname`
mkdir -p /root/working/experiments/training

/root/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8
-dont-zip -first-step 7 -last-step 7 -external-bin-dir
/root/mosesdecoder/tools -f en -e de -alignment grow-diag-final-and
-reordering msd-bidirectional-fe -extract-file
/root/working/experiments/model/extract.2 -reordering-table
/root/working/experiments/model/reordering-table.3

echo 'finished at '`date`
touch /root/working/experiments/steps/3/TRAINING_build-reordering.3.DONE

==> TRAINING_build-reordering.3.INFO <==
lexicalized-reordering = msd-bidirectional-fe
INPUT = USED /root/working/experiments/model/extract.2
# reuse run 2 for TRAINING:extract-phrases

==> TRAINING_build-reordering.3.STDERR <==
Using SCRIPTS_ROOTDIR: /root/mosesdecoder/scripts
using gzip
(7) learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.1) [no factors] learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.2) building tables @ Tue Mar 28 12:19:55 CEST 2017
Executing: /root/mosesdecoder/scripts/../bin/lexical-reordering-score
/root/working/experiments/model/extract.2.o.sorted.gz 0.5
/root/working/experiments/model/reordering-table.3. --model "wbe msd
wbe-msd-bidirectional-fe"
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical,
phrase-based and word-based-extraction

==> TRAINING_build-reordering.3.STDOUT <==
starting at Di 28. Mär 12:19:54 CEST 2017 on machine-VirtualBox

==> TRAINING_build-reordering.3.STDERR <==
terminate called after throwing an instance of 'FileFormatException'
  what():  phrase-extract/lexical-reordering/score.cpp:260 in void
split_line(const StringPiece&, StringPiece&, StringPiece&, StringPiece&,
StringPiece&, StringPiece&, float&) threw FileFormatException because
`errIndex == next.data()'.
Invalid extract file format: inhaled corticosteroids ||| inhalative
Kortikoichaemic ||| ischämische ||| mono other
Aborted (core dumped)
Exit code: 134
*ERROR: Lexical reordering scoring failed at
/root/mosesdecoder/scripts/training/train-model.perl line 1924.*

Does someone know how to get rid of that problem?

Thanks in advance and best regards,
Anna

-- 
Anna Kupś
+48 782 823 850 | www.linkedin.com/in/annakups
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error: Lexical reordering scoring failed

[James]

Hi,

I'm training a translation model on Ubuntu with mgiza, nplm but somehow it
failed at the lexical reordering. Could you help me out what's going on
here? Thanks!

The system description:
---
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
---

Here is the command I used:
---
"nohup nice /home/ubuntu/m/mosesdecoder/scripts/training/train-model.perl
--root-dir ./training --parallel --model-dir ./training/model --mgiza
--mgiza-cpus 8 --corpus /home/ubuntu/m/data/train-clean --external-bin-dir
/home/ubuntu/m/mosesdecoder/tools --f ja --e ko --alignment
grow-diag-final-and --reordering msd-bidirectional-fe --score-options
"--GoodTuring" --lm 0:5:/home/ubuntu/m/nn/kolm.nnlm:8 --cores 16
--sort-buffer-size 10G >& training.out &"
---

And, the error log snippet from the "training.out" file generated:
---

H333444 Training Finished at: Tue Aug  9 07:49:18 2016

Entire Viterbi H333444 Training took: 1814 seconds

==


Entire Training took: 3950 seconds

Program Finished at: Tue Aug  9 07:49:18 2016


==

Merging A3.final.part* tables

Executing: /home/ubuntu/m/mosesdecoder/tools/merge_alignment.py
/home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final.part*>/home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final

Combined 8 files, totally 780579 sents

Executing: rm -f /home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final.gz

Executing: gzip /home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final

Waiting for second GIZA process...

(3) generate word alignment @ Tue Aug  9 07:49:46 UTC 2016

Combining forward and inverted alignment from files:

  /home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final.{bz2,gz}

  /home/ubuntu/m/training/giza.ko-ja/ko-ja.A3.final.{bz2,gz}

Executing: mkdir -p /home/ubuntu/m/training/model

Executing: /home/ubuntu/m/mosesdecoder/scripts/training/giza2bal.pl -d
"gzip -cd /home/ubuntu/m/training/giza.ko-ja/ko-ja.A3.final.gz" -i "gzip
-cd /home/ubuntu/m/training/giza.ja-ko/ja-ko.A3.final.gz"
|/home/ubuntu/m/mosesdecoder/scripts/../bin/symal -alignment="grow"
-diagonal="yes" -final="yes" -both="yes" >
/home/ubuntu/m/training/model/aligned.grow-diag-final-and

symal: computing grow alignment: diagonal (1) final (1)both-uncovered (1)

skip=<0> counts=<780579>

(4) generate lexical translation table 0-0 @ Tue Aug  9 07:51:13 UTC 2016

(/home/ubuntu/m/data/train-clean.ja,/home/ubuntu/m/data/train-clean.ko,/home/ubuntu/m/training/model/lex)

!

Saved: /home/ubuntu/m/training/model/lex.f2e and
/home/ubuntu/m/training/model/lex.e2f

FILE: /home/ubuntu/m/data/train-clean.ko

FILE: /home/ubuntu/m/data/train-clean.ja

FILE: /home/ubuntu/m/training/model/aligned.grow-diag-final-and

(5) extract phrases @ Tue Aug  9 07:52:16 UTC 2016

/home/ubuntu/m/mosesdecoder/scripts/generic/extract-parallel.perl 16 split
"sort -S 10G   " /home/ubuntu/m/mosesdecoder/scripts/../bin/extract
/home/ubuntu/m/data/train-clean.ko /home/ubuntu/m/data/train-clean.ja
/home/ubuntu/m/training/model/aligned.grow-diag-final-and
/home/ubuntu/m/training/model/extract 7 orientation --model wbe-msd
--GZOutput

Executing:
/home/ubuntu/m/mosesdecoder/scripts/generic/extract-parallel.perl 16 split
"sort -S 10G   " /home/ubuntu/m/mosesdecoder/scripts/../bin/extract
/home/ubuntu/m/data/train-clean.ko /home/ubuntu/m/data/train-clean.ja
/home/ubuntu/m/training/model/aligned.grow-diag-final-and
/home/ubuntu/m/training/model/extract 7 orientation --model wbe-msd
--GZOutput

MAX 7 1 0

Started Tue Aug  9 07:52:16 2016

using gzip

isBSDSplit=0

Executing: mkdir -p /home/ubuntu/m/training/model/tmp.9146; ls -l
/home/ubuntu/m/training/model/tmp.9146

total=780579 line-per-split=48787

split -d -l 48787 -a 7 /home/ubuntu/m/data/train-clean.ko
/home/ubuntu/m/training/model/tmp.9146/target.split -d -l 48787 -a 7
/home/ubuntu/m/data/train-clean.ja
/home/ubuntu/m/training/model/tmp.9146/source.split -d -l 48787 -a 7
/home/ubuntu/m/training/model/aligned.grow-diag-final-and
/home/ubuntu/m/training/model/tmp.9146/align.merging extract / extract.inv

gunzip -c /home/ubuntu/m/training/model/tmp.9146/extract.000.gz

[Moses-support] ERROR: Lexical reordering scoring failed at /home/berry/mosesdecoder/scripts/training/train-model.perl line 1922.

[BIRENDRA CHAUHAN SINGH]

on running below command for training the baseline model.I have attached
the output result file below.It shows all the error.

nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
 -corpus ~/corpus/news-commentary-v8.fr-en.clean \
 -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe \
 -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8  \
 -external-bin-dir ~/mosesdecoder/tools >& training.out &

Am getting erros like :

gzip: gzip: /home/berry/working/train/model/tmp.25178/extract.000.inv.gz:
No such file or directory
/home/berry/working/train/model/tmp.25178/extract.000.gz: No such
file or directory

ERROR: Scoring of phrases failed at
/home/berry/mosesdecoder/scripts/training/train-model.perl line 1784.

/home/berry/mosesdecoder/scripts/../bin/consolidate: error while
loading shared libraries: libboost_serialization.so.1.59.0: cannot
open shared object file: No such file or directory

ERROR: Lexical reordering scoring failed at
/home/berry/mosesdecoder/scripts/training/train-model.perl line 1922.


training.out
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed

[Philipp Koehn]

Hi,

you are building a hierarchical phrase-based model,
so you cannot have a lexicalized reordering model.

Remove -reordering msd-bidirectional-fe from your
command line.

-phi

On Mon, Jan 5, 2015 at 12:51 AM, Jon Olds joft...@yahoo.co.uk wrote:
 Hi,

 I’m trying to build a hierarchical model using the same (cleaned) data
 that I have used successfully to build a phrase model and I keep getting
 the following error:

 ERROR: Lexical reordering scoring failed at
 /home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.

 The command I used was:

 nohup nice ~/tools/mosesdecoder/scripts/training/train-model.perl
 -root-dir train -corpus ~/data/tok/base.clean -f fr -e en -alignment
 grow-diag-final-and -reordering msd-bidirectional-fe -lm
 0:3:$HOME/data/lm/base.blm.en:8 -external-bin-dir ~/tools/trainingtools
 -hierarchical -glue-grammar -max-phrase-length 5  training.out 

 Any ideas?

 Many thanks,

 Jon

 /home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1
 sort /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
 /home/ubuntu/working/train/model/extract.inv.sorted.gz
 /home/ubuntu/working/train/model/lex.e2f
 /home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse
 --Hierarchical 1
 Executing:
 /home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1
 sort /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
 /home/ubuntu/working/train/model/extract.inv.sorted.gz
 /home/ubuntu/working/train/model/lex.e2f
 /home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse
 --Hierarchical 1
 Started Sun Jan  4 12:55:12 2015
 ln -s /home/ubuntu/working/train/model/extract.inv.sorted.gz
 /home/ubuntu/working/train/model/tmp.25446/extract.0.gz
 /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
 /home/ubuntu/working/train/model/tmp.25446/extract.0.gz
 /home/ubuntu/working/train/model/lex.e2f
 /home/ubuntu/working/train/model/tmp.25446/phrase-table.half.000.gz
 --Inverse --Hierarchical  2 /dev/stderr
 /home/ubuntu/working/train/model/tmp.25446/run.0.shgunzip -c
 /home/ubuntu/working/train/model/tmp.25446/phrase-table.half.*.gz 2
 /dev/stderr| LC_ALL=C sort -T
 /home/ubuntu/working/train/model/tmp.25446  | gzip -c 
 /home/ubuntu/working/train/model/rule-table.half.e2f.gz  2 /dev/stderr
 rm -rf /home/ubuntu/working/train/model/tmp.25446
 Finished Sun Jan  4 13:15:56 2015
 (6.6) consolidating the two halves @ Sun Jan  4 13:15:56 UTC 2015
 Executing: /home/ubuntu/tools/mosesdecoder/scripts/../bin/consolidate
 /home/ubuntu/working/train/model/rule-table.half.f2e.gz
 /home/ubuntu/working/train/model/rule-table.half.e2f.gz /dev/stdout
 --Hierarchical | gzip -c  /home/ubuntu/working/train/model/rule-table.gz
 Consolidate v2.0 written by Philipp Koehn
 consolidating direct and indirect rule tables
 processing hierarchical rules
 Executing: rm -f /home/ubuntu/working/train/model/rule-table.half.*
 (7) learn reordering model @ Sun Jan  4 13:25:58 UTC 2015
 (7.1) [no factors] learn reordering model @ Sun Jan  4 13:25:58 UTC 2015
 (7.2) building tables @ Sun Jan  4 13:25:58 UTC 2015
 Executing:
 /home/ubuntu/tools/mosesdecoder/scripts/../bin/lexical-reordering-score
 /home/ubuntu/working/train/model/extract.o.sorted.gz 0.5
 /home/ubuntu/working/train/model/reordering-table. --model wbe msd
 wbe-msd-bidirectional-fe
 Lexical Reordering Scorer
 scores lexical reordering models of several types (hierarchical,
 phrase-based and word-based-extraction
 terminate called after throwing an instance of 'util::ErrnoException'
what():  util/file.cc:68 in int util::OpenReadOrThrow(const char*)
 threw ErrnoException because `-1 == (ret = open(name, 00))'.
 No such file or directory while opening
 /home/ubuntu/working/train/model/extract.o.sorted.gz
 Aborted (core dumped)
 Exit code: 134
 ERROR: Lexical reordering scoring failed at
 /home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Lexical reordering scoring failed

[Jon Olds]

Hi,

I’m trying to build a hierarchical model using the same (cleaned) data 
that I have used successfully to build a phrase model and I keep getting 
the following error:

ERROR: Lexical reordering scoring failed at 
/home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.

The command I used was:

nohup nice ~/tools/mosesdecoder/scripts/training/train-model.perl 
-root-dir train -corpus ~/data/tok/base.clean -f fr -e en -alignment 
grow-diag-final-and -reordering msd-bidirectional-fe -lm 
0:3:$HOME/data/lm/base.blm.en:8 -external-bin-dir ~/tools/trainingtools 
-hierarchical -glue-grammar -max-phrase-length 5  training.out 

Any ideas?

Many thanks,

Jon

/home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1 
sort /home/ubuntu/tools/mosesdecoder/scripts/../bin/score 
/home/ubuntu/working/train/model/extract.inv.sorted.gz 
/home/ubuntu/working/train/model/lex.e2f 
/home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse 
--Hierarchical 1
Executing: 
/home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1 
sort /home/ubuntu/tools/mosesdecoder/scripts/../bin/score 
/home/ubuntu/working/train/model/extract.inv.sorted.gz 
/home/ubuntu/working/train/model/lex.e2f 
/home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse 
--Hierarchical 1
Started Sun Jan  4 12:55:12 2015
ln -s /home/ubuntu/working/train/model/extract.inv.sorted.gz 
/home/ubuntu/working/train/model/tmp.25446/extract.0.gz
/home/ubuntu/tools/mosesdecoder/scripts/../bin/score 
/home/ubuntu/working/train/model/tmp.25446/extract.0.gz 
/home/ubuntu/working/train/model/lex.e2f 
/home/ubuntu/working/train/model/tmp.25446/phrase-table.half.000.gz 
--Inverse --Hierarchical  2 /dev/stderr
/home/ubuntu/working/train/model/tmp.25446/run.0.shgunzip -c 
/home/ubuntu/working/train/model/tmp.25446/phrase-table.half.*.gz 2 
/dev/stderr| LC_ALL=C sort -T 
/home/ubuntu/working/train/model/tmp.25446  | gzip -c  
/home/ubuntu/working/train/model/rule-table.half.e2f.gz  2 /dev/stderr 
rm -rf /home/ubuntu/working/train/model/tmp.25446
Finished Sun Jan  4 13:15:56 2015
(6.6) consolidating the two halves @ Sun Jan  4 13:15:56 UTC 2015
Executing: /home/ubuntu/tools/mosesdecoder/scripts/../bin/consolidate 
/home/ubuntu/working/train/model/rule-table.half.f2e.gz 
/home/ubuntu/working/train/model/rule-table.half.e2f.gz /dev/stdout 
--Hierarchical | gzip -c  /home/ubuntu/working/train/model/rule-table.gz
Consolidate v2.0 written by Philipp Koehn
consolidating direct and indirect rule tables
processing hierarchical rules
Executing: rm -f /home/ubuntu/working/train/model/rule-table.half.*
(7) learn reordering model @ Sun Jan  4 13:25:58 UTC 2015
(7.1) [no factors] learn reordering model @ Sun Jan  4 13:25:58 UTC 2015
(7.2) building tables @ Sun Jan  4 13:25:58 UTC 2015
Executing: 
/home/ubuntu/tools/mosesdecoder/scripts/../bin/lexical-reordering-score 
/home/ubuntu/working/train/model/extract.o.sorted.gz 0.5 
/home/ubuntu/working/train/model/reordering-table. --model wbe msd 
wbe-msd-bidirectional-fe
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical, 
phrase-based and word-based-extraction
terminate called after throwing an instance of 'util::ErrnoException'
   what():  util/file.cc:68 in int util::OpenReadOrThrow(const char*) 
threw ErrnoException because `-1 == (ret = open(name, 00))'.
No such file or directory while opening 
/home/ubuntu/working/train/model/extract.o.sorted.gz
Aborted (core dumped)
Exit code: 134
ERROR: Lexical reordering scoring failed at 
/home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Lexical reordering scoring failed

[Tom Hoar]

I experienced an error that I've not seen before using train-model.perl 
from Moses Release 1.0. Here are the final lines of the output log as 
the script terminated. The error message Illegal reordering type used: 
d is new to me.


(7) learn reordering model @ Sun Jun 1 00:12:55 IST 2014
(7.1) [no factors] learn reordering model @ Sun Jun 1 00:12:55 IST 2014
(7.2) building tables @ Sun Jun 1 00:12:55 IST 2014
Executing: 
/usr/local/lib/mosesdecoder/scripts/../bin/lexical-reordering-score 
/opt/domy/TRAININGS/alignments/align-td_fr_ui-en_us-fr_fr/giza.extract.en_us-fr_fr/ext.7-gram.o.sorted.gz 
0.5 
/opt/domy/ENGINES/tables/phrase-s=en_us-t=fr_fr-p=td_fr_ui-a=giza-g=7/reordering-table. 
--model wbe msd wbe-msd-bidirectional-fe

Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical, 
phrase-based and word-based-extraction

*Illegal reordering type used: d*
Exit code: 1
ERROR: Lexical reordering scoring failed at 
/usr/local/bin/train-model.perl line 1705.



I suspect two potential causes.

1) the system only has about 50 GB of free hard drive space. There are 
several tools in the training toolchain that create atomic temp files 
that disappear immediately on an error. Is it possible this is one of 
those place, and would this error occur if the system ran out of hard 
drive space during this stage? I.e. it needs more than 50GB temp file 
space?


2) the log has several warning messages that sentence pairs violate 9:1 
ratio. E.g.


WARNING: The following sentence pair has source/target sentence length 
ration more than the maximum allowed limit for a source word fertility

source length = 1 target length = 15 ratio 15 ferility limit : 9
Shortening sentence
Sent No: 212107 , No. Occurrences: 1

This should not be happening with our corpus preparation tools. There 
doesn't seem to be anything particularly different about this training 
corpus. So, we will look for our leak that let them pass our 
preparation filters. Nonetheless, would these ratio violations cause the 
fatal error in step 7 as logged above?


Can anyone can confirm my suspensions or suggest an alternate cause.

Thanks,
Tom
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[李惠惠]

Hi Barry,
Thanks for the instructions.
I check the tmp directory and it seems that the align. file is broken.
And I rerun the extract-parallel perl separately but it doesn't work either.
Here is the link to my small corpus.
https://www.dropbox.com/sh/beffab8g215awz6/2dCp0G_pdj

Thank you,
Hui

2014-02-24 18:46 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk:

 Hi Hui

 I have a vague memory that very short corpora cause problems ...

 If you edit scripts/generic/extract-parallel.perl to comment out the
 following line (line 223 in my version):

 $cmd = rm -rf $TMPDIR \n;

 then rerun, it should retain the tmp directory created during
 extraction. You could then rerun extract manually and look for the
 problem. If you don't see anything odd, then upload to pastebin of
 dropbox and send out the link - or just email if it's small.

 cheers - Barry


 On 24/02/14 16:00, 李惠惠 wrote:
  Hi Barry,
  I tried to use absolute paths like this:
 
  hli@themisto:~$ /home/hli/MT/moses/scripts/training/train-model.perl
  /home/hli/MT -corpus /home/hli/MT/lm/1545.test.lowercased
  -external-bin-dir /home/hli/MT/bin -e de -f ade -alignment
  grow-diag-final-and -reordering msd-bidirectional-fe -lm
  0:3:/home/hli/MT/lm/1545.test.lm  training.out 
 
  but the same error still exists.
 
  Is that anything wrong with my corpus?
 
  Best,
  Hui
 
 
  2014-02-24 16:48 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk
  mailto:bhad...@staffmail.ed.ac.uk:
 
  Hi Hui
 
  It looks like your extract failed, but there is no error message.
  Do you think it ran out of disk? Could you try again, but use
  absolute paths everywhere?
 
  cheers - Barry
 
 

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[Barry Haddow]

Hi Hui

What do you mean by broken? I meant for you to upload the contents of
the tmp directory to dropbox,

cheers - Barry

On 25/02/14 15:54, 李惠惠 wrote:
 Hi Barry,
 Thanks for the instructions.
 I check the tmp directory and it seems that the align. file is broken.
 And I rerun the extract-parallel perl separately but it doesn't work
 either.
 Here is the link to my small corpus.
 https://www.dropbox.com/sh/beffab8g215awz6/2dCp0G_pdj

 Thank you,
 Hui

 2014-02-24 18:46 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk:

 Hi Hui

 I have a vague memory that very short corpora cause problems ...

 If you edit scripts/generic/extract-parallel.perl to comment out the
 following line (line 223 in my version):

 $cmd = rm -rf $TMPDIR \n;

 then rerun, it should retain the tmp directory created during
 extraction. You could then rerun extract manually and look for the
 problem. If you don't see anything odd, then upload to pastebin of
 dropbox and send out the link - or just email if it's small.

 cheers - Barry


 On 24/02/14 16:00, 李惠惠 wrote:
  Hi Barry,
  I tried to use absolute paths like this:
 
  hli@themisto:~$ /home/hli/MT/moses/scripts/training/train-model.perl
  /home/hli/MT -corpus /home/hli/MT/lm/1545.test.lowercased
  -external-bin-dir /home/hli/MT/bin -e de -f ade -alignment
  grow-diag-final-and -reordering msd-bidirectional-fe -lm
  0:3:/home/hli/MT/lm/1545.test.lm  training.out 
 
  but the same error still exists.
 
  Is that anything wrong with my corpus?
 
  Best,
  Hui
 
 
  2014-02-24 16:48 GMT+01:00 Barry Haddow
 bhad...@staffmail.ed.ac.uk mailto:bhad...@staffmail.ed.ac.uk
  mailto:bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk:
 
  Hi Hui
 
  It looks like your extract failed, but there is no error message.
  Do you think it ran out of disk? Could you try again, but use
  absolute paths everywhere?
 
  cheers - Barry
 
 


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[Barry Haddow]

Hi Hui

Something went wrong earlier in training which meant that 
extract.o.sorted.gz was not created. You need to attach the log of 
train-model,

cheers - Barry

On 24/02/14 08:38, 李惠惠 wrote:



 Hi all,

 when I was training the phrase model,
 hli@themisto:~/MT$ moses/scripts/training/train-model.perl home/hli/MT 
 -corpus lm/1545.test.lowercased -external-bin-dir bin -e de -f ade 
 -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 
 0:3:/home/hli/MT/lm/1545.test.lm

 I obtained this error:

 scores lexical reordering models of several types (hierarchical, 
 phrase-based and word-based-extraction
 terminate called after throwing an instance of 'util::ErrnoException'
   what():  util/file.cc:67 in int util::OpenReadOrThrow(const char*) 
 threw ErrnoException because `-1 == (ret = open(name, 00))'.
 No such file or directory while opening ./model/extract.o.sorted.gz
 Aborted (core dumped)
 Exit code: 134
 ERROR: Lexical reordering scoring failed at 
 moses/scripts/training/train-model.perl line 1738.

 Attached is my corpus, could you help me to find out what's wrong with 
 it ?

 Thx a lot!

 Best Regards,
 Hui Li
 Institute of Computer Science
 Heidelberg

 *
 *





 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[Barry Haddow]

Hi Hui

It looks like your extract failed, but there is no error message. Do you 
think it ran out of disk? Could you try again, but use absolute paths 
everywhere?

cheers - Barry

On 24/02/14 09:44, 李惠惠 wrote:


 Hi Barry,
 attached is my log of phrase model training. Thx a lot.

 Best,
 Hui



 2014-02-24 10:18 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk 
 mailto:bhad...@staffmail.ed.ac.uk:

 Hi Hui

 Something went wrong earlier in training which meant that
 extract.o.sorted.gz was not created. You need to attach the log of
 train-model,

 cheers - Barry


 On 24/02/14 08:38, 李惠惠 wrote:




 Hi all,

 when I was training the phrase model,
 hli@themisto:~/MT$ moses/scripts/training/train-model.perl
 home/hli/MT -corpus lm/1545.test.lowercased -external-bin-dir
 bin -e de -f ade -alignment grow-diag-final-and -reordering
 msd-bidirectional-fe -lm 0:3:/home/hli/MT/lm/1545.test.lm

 I obtained this error:

 scores lexical reordering models of several types
 (hierarchical, phrase-based and word-based-extraction
 terminate called after throwing an instance of
 'util::ErrnoException'
   what():  util/file.cc:67 in int util::OpenReadOrThrow(const
 char*) threw ErrnoException because `-1 == (ret = open(name,
 00))'.
 No such file or directory while opening
 ./model/extract.o.sorted.gz
 Aborted (core dumped)
 Exit code: 134
 ERROR: Lexical reordering scoring failed at
 moses/scripts/training/train-model.perl line 1738.

 Attached is my corpus, could you help me to find out what's
 wrong with it ?

 Thx a lot!

 Best Regards,
 Hui Li
 Institute of Computer Science
 Heidelberg

 *
 *





 ___
 Moses-support mailing list
 Moses-support@mit.edu mailto:Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support



 -- 
 The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.





 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[李惠惠]

Hi Barry,
I tried to use absolute paths like this:

hli@themisto:~$ /home/hli/MT/moses/scripts/training/train-model.perl
/home/hli/MT -corpus /home/hli/MT/lm/1545.test.lowercased -external-bin-dir
/home/hli/MT/bin -e de -f ade -alignment grow-diag-final-and -reordering
msd-bidirectional-fe -lm 0:3:/home/hli/MT/lm/1545.test.lm  training.out 

but the same error still exists.

Is that anything wrong with my corpus?

Best,
Hui


2014-02-24 16:48 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk:

 Hi Hui

 It looks like your extract failed, but there is no error message. Do you
 think it ran out of disk? Could you try again, but use absolute paths
 everywhere?

 cheers - Barry


 On 24/02/14 09:44, 李惠惠 wrote:



 Hi Barry,
 attached is my log of phrase model training. Thx a lot.

 Best,
 Hui



 2014-02-24 10:18 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.ukmailto:
 bhad...@staffmail.ed.ac.uk:


 Hi Hui

 Something went wrong earlier in training which meant that
 extract.o.sorted.gz was not created. You need to attach the log of
 train-model,

 cheers - Barry


 On 24/02/14 08:38, 李惠惠 wrote:




 Hi all,

 when I was training the phrase model,
 hli@themisto:~/MT$ moses/scripts/training/train-model.perl
 home/hli/MT -corpus lm/1545.test.lowercased -external-bin-dir
 bin -e de -f ade -alignment grow-diag-final-and -reordering
 msd-bidirectional-fe -lm 0:3:/home/hli/MT/lm/1545.test.lm

 I obtained this error:

 scores lexical reordering models of several types
 (hierarchical, phrase-based and word-based-extraction
 terminate called after throwing an instance of
 'util::ErrnoException'
   what():  util/file.cc:67 in int util::OpenReadOrThrow(const
 char*) threw ErrnoException because `-1 == (ret = open(name,
 00))'.
 No such file or directory while opening
 ./model/extract.o.sorted.gz
 Aborted (core dumped)
 Exit code: 134
 ERROR: Lexical reordering scoring failed at
 moses/scripts/training/train-model.perl line 1738.

 Attached is my corpus, could you help me to find out what's
 wrong with it ?

 Thx a lot!

 Best Regards,
 Hui Li
 Institute of Computer Science
 Heidelberg

 *
 *





 ___
 Moses-support mailing list
 Moses-support@mit.edu mailto:Moses-support@mit.edu

 http://mailman.mit.edu/mailman/listinfo/moses-support



 -- The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.





 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support



 --
 The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed at moses/scripts/training/train-model.perl line 1738.

[Barry Haddow]

Hi Hui

I have a vague memory that very short corpora cause problems ...

If you edit scripts/generic/extract-parallel.perl to comment out the
following line (line 223 in my version):

$cmd = rm -rf $TMPDIR \n;

then rerun, it should retain the tmp directory created during
extraction. You could then rerun extract manually and look for the
problem. If you don't see anything odd, then upload to pastebin of
dropbox and send out the link - or just email if it's small.

cheers - Barry


On 24/02/14 16:00, 李惠惠 wrote:
 Hi Barry,
 I tried to use absolute paths like this:

 hli@themisto:~$ /home/hli/MT/moses/scripts/training/train-model.perl
 /home/hli/MT -corpus /home/hli/MT/lm/1545.test.lowercased
 -external-bin-dir /home/hli/MT/bin -e de -f ade -alignment
 grow-diag-final-and -reordering msd-bidirectional-fe -lm
 0:3:/home/hli/MT/lm/1545.test.lm  training.out 

 but the same error still exists.

 Is that anything wrong with my corpus?

 Best,
 Hui


 2014-02-24 16:48 GMT+01:00 Barry Haddow bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk:

 Hi Hui

 It looks like your extract failed, but there is no error message.
 Do you think it ran out of disk? Could you try again, but use
 absolute paths everywhere?

 cheers - Barry


 On 24/02/14 09:44, 李惠惠 wrote:



 Hi Barry,
 attached is my log of phrase model training. Thx a lot.

 Best,
 Hui



 2014-02-24 10:18 GMT+01:00 Barry Haddow
 bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk
 mailto:bhad...@staffmail.ed.ac.uk:


 Hi Hui

 Something went wrong earlier in training which meant that
 extract.o.sorted.gz was not created. You need to attach the log of
 train-model,

 cheers - Barry


 On 24/02/14 08:38, 李惠惠 wrote:




 Hi all,

 when I was training the phrase model,
 hli@themisto:~/MT$ moses/scripts/training/train-model.perl
 home/hli/MT -corpus lm/1545.test.lowercased -external-bin-dir
 bin -e de -f ade -alignment grow-diag-final-and -reordering
 msd-bidirectional-fe -lm 0:3:/home/hli/MT/lm/1545.test.lm

 I obtained this error:

 scores lexical reordering models of several types
 (hierarchical, phrase-based and word-based-extraction
 terminate called after throwing an instance of
 'util::ErrnoException'
 what(): util/file.cc:67 in int util::OpenReadOrThrow(const
 char*) threw ErrnoException because `-1 == (ret = open(name,
 00))'.
 No such file or directory while opening
 ./model/extract.o.sorted.gz
 Aborted (core dumped)
 Exit code: 134
 ERROR: Lexical reordering scoring failed at
 moses/scripts/training/train-model.perl line 1738.

 Attached is my corpus, could you help me to find out what's
 wrong with it ?

 Thx a lot!

 Best Regards,
 Hui Li
 Institute of Computer Science
 Heidelberg

 *
 *





 ___
 Moses-support mailing list
 Moses-support@mit.edu mailto:Moses-support@mit.edu
 mailto:Moses-support@mit.edu mailto:Moses-support@mit.edu

 http://mailman.mit.edu/mailman/listinfo/moses-support



 -- The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.





 ___
 Moses-support mailing list
 Moses-support@mit.edu mailto:Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support



 -- 
 The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.



The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support