[Moses-support] Segmentation Fault on tuning phase
Hi, I've just pulled the code and rebuild the MT system and I'm getting a segmentation fault during tuning step using EMS in a experiment that was working well with a version of the code pulled on October 2. I have pulled and refreshed the code because I was facing some problems getting the word alignments with moses --server and I saw that the code that deals with the server params has been updated recently. The model is a factored model and the error in the log is: Using SCRIPTS_ROOTDIR: /opt/moses/scripts Asking moses for feature names and values from /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights exec: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights > ./features.list 2> /dev/null MERT starting values and ranges for random generation: LexicalReordering0 = 0.300 ( 0.00 .. 1.00) LexicalReordering0 = 0.300 ( 0.00 .. 1.00) LexicalReordering0 = 0.300 ( 0.00 .. 1.00) LexicalReordering0 = 0.300 ( 0.00 .. 1.00) LexicalReordering0 = 0.300 ( 0.00 .. 1.00) LexicalReordering0 = 0.300 ( 0.00 .. 1.00) Distortion0 = 0.300 ( 0.00 .. 1.00) LM0 = 0.500 ( 0.00 .. 1.00) LM1 = 0.500 ( 0.00 .. 1.00) LM2 = 0.500 ( 0.00 .. 1.00) WordPenalty0 = -1.000 ( 0.00 .. 1.00) PhrasePenalty0 = 0.200 ( 0.00 .. 1.00) TranslationModel0 = 0.200 ( 0.00 .. 1.00) TranslationModel0 = 0.200 ( 0.00 .. 1.00) TranslationModel0 = 0.200 ( 0.00 .. 1.00) TranslationModel0 = 0.200 ( 0.00 .. 1.00) TranslationModel1 = 0.200 ( 0.00 .. 1.00) TranslationModel1 = 0.200 ( 0.00 .. 1.00) TranslationModel1 = 0.200 ( 0.00 .. 1.00) TranslationModel1 = 0.200 ( 0.00 .. 1.00) TranslationModel2 = 0.200 ( 0.00 .. 1.00) TranslationModel2 = 0.200 ( 0.00 .. 1.00) TranslationModel2 = 0.200 ( 0.00 .. 1.00) TranslationModel2 = 0.200 ( 0.00 .. 1.00) GenerationModel0 = 0.300 ( 0.00 .. 1.00) GenerationModel0 = 0.000 ( 0.00 .. 1.00) GenerationModel1 = 0.300 ( 0.00 .. 1.00) GenerationModel1 = 0.000 ( 0.00 .. 1.00) featlist: LexicalReordering0=0.30 featlist: LexicalReordering0=0.30 featlist: LexicalReordering0=0.30 featlist: LexicalReordering0=0.30 featlist: LexicalReordering0=0.30 featlist: LexicalReordering0=0.30 featlist: Distortion0=0.30 featlist: LM0=0.50 featlist: LM1=0.50 featlist: LM2=0.50 featlist: WordPenalty0=-1.00 featlist: PhrasePenalty0=0.20 featlist: TranslationModel0=0.20 featlist: TranslationModel0=0.20 featlist: TranslationModel0=0.20 featlist: TranslationModel0=0.20 featlist: TranslationModel1=0.20 featlist: TranslationModel1=0.20 featlist: TranslationModel1=0.20 featlist: TranslationModel1=0.20 featlist: TranslationModel2=0.20 featlist: TranslationModel2=0.20 featlist: TranslationModel2=0.20 featlist: TranslationModel2=0.20 featlist: GenerationModel0=0.30 featlist: GenerationModel0=0.00 featlist: GenerationModel1=0.30 featlist: GenerationModel1=0.00 Saved: ./run1.moses.ini Normalizing lambdas: 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 0.50 0.50 -1.00 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.30 0.00 0.30 0.00 DECODER_CFG = -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.00 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.00' Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.00 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.00' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.00 TranslationModel1= 0.025641 0.025641 0.025641
Re: [Moses-support] Segmentation Fault during Tuning
Hi, with this modification it works Thanks a lot Alex El 12 oct 2015 a las 09:09, Philipp Koehn <p...@jhu.edu> escribió: Hi, in t2, you do generate an output lemma factor - which may be the cause of this problem (even though you do not seem to use the output lemma anywhere else). Does it still core dump, if you change translation factors to: translation-factors = "lemma -> lemma, pos -> pos, word -> word + lemma + pos" -phi On Sat, Oct 10, 2015 at 9:52 AM, Alex Martinez <cmx...@me.com> wrote: Hello, I'm trying to build a factored system using EMS based on this example from the tutorial: - % train-model.perl \ --corpus factored-corpus/proj-syndicate.1000 \ --root-dir morphgen-backoff \ --f de --e en \ --lm 0:3:factored-corpus/surface.lm:0 \ --lm 2:3:factored-corpus/pos.lm:0 \ --translation-factors 1-1+3-2+0-0,2 \ --generation-factors 1-2+1,2-0 \ --decoding-steps t0,g0,t1,g1:t2 \ --external-bin-dir .../tools -- I'm getting a segmentation fault during tuning and I have the feeling that the problem is related to the line defining the decoding-steps. What I have on my EMS config file to get a similar model is: ### factored training: specify here which factors used # if none specified, single factor training is assumed # (one translation step, surface to surface) # input-factors = word lemma pos output-factors = word lemma pos alignment-factors = "word+lemma -> word+lemma" translation-factors = "lemma -> lemma, pos -> pos, word -> word + pos" reordering-factors = "word -> word" generation-factors = "lemma -> pos, lemma+pos -> word" decoding-steps = "t0,g0,t1,g1:t2" generation-type = single prune-generation = "$moses-bin-dir/pruneGeneration 100" - The training fails in the tuning step and I'm getting this in the TUNING_tune.1.STDERR: Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/processfin/model/moses.bin.ini.1 -weight-overwrite 'WordPenalty0= -0.128205 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM2= 0.064103 LM0= 0.064103 GenerationModel1= 0.038462 0.00 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel0= 0.038462 PhrasePenalty0= 0.025641 Distortion0= 0.038462 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM1= 0.064103' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out Segmentation fault (core dumped) Exit code: 139 The decoder died. CONFIG WAS -weight-overwrite 'WordPenalty0= -0.128205 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM2= 0.064103 LM0= 0.064103 GenerationModel1= 0.038462 0.00 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel0= 0.038462 PhrasePenalty0= 0.025641 Distortion0= 0.038462 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM1= 0.064103' cp: cannot stat ‘/mnt/a62/devel/en_es/processfin/tuning/tmp.1/moses.ini’: No such file or directory --- If I change this line in the config file from decoding-steps = "t0,g0,t1,g1:t2" to decoding-steps = "t0,g0,t1,g1" then the training ends without errors. I'll appreciate suggestions on how to solve that. Alex ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Segmentation Fault during Tuning
Hello, I'm trying to build a factored system using EMS based on this example from the tutorial: - % train-model.perl \ --corpus factored-corpus/proj-syndicate.1000 \ --root-dir morphgen-backoff \ --f de --e en \ --lm 0:3:factored-corpus/surface.lm:0 \ --lm 2:3:factored-corpus/pos.lm:0 \ --translation-factors 1-1+3-2+0-0,2 \ --generation-factors 1-2+1,2-0 \ --decoding-steps t0,g0,t1,g1:t2 \ --external-bin-dir .../tools -- I'm getting a segmentation fault during tuning and I have the feeling that the problem is related to the line defining the decoding-steps. What I have on my EMS config file to get a similar model is: ### factored training: specify here which factors used # if none specified, single factor training is assumed # (one translation step, surface to surface) # input-factors = word lemma pos output-factors = word lemma pos alignment-factors = "word+lemma -> word+lemma" translation-factors = "lemma -> lemma, pos -> pos, word -> word + pos" reordering-factors = "word -> word" generation-factors = "lemma -> pos, lemma+pos -> word" decoding-steps = "t0,g0,t1,g1:t2" generation-type = single prune-generation = "$moses-bin-dir/pruneGeneration 100" - The training fails in the tuning step and I'm getting this in the TUNING_tune.1.STDERR: Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/processfin/model/moses.bin.ini.1 -weight-overwrite 'WordPenalty0= -0.128205 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM2= 0.064103 LM0= 0.064103 GenerationModel1= 0.038462 0.00 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel0= 0.038462 PhrasePenalty0= 0.025641 Distortion0= 0.038462 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM1= 0.064103' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out Segmentation fault (core dumped) Exit code: 139 The decoder died. CONFIG WAS -weight-overwrite 'WordPenalty0= -0.128205 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM2= 0.064103 LM0= 0.064103 GenerationModel1= 0.038462 0.00 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel0= 0.038462 PhrasePenalty0= 0.025641 Distortion0= 0.038462 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM1= 0.064103' cp: cannot stat ‘/mnt/a62/devel/en_es/processfin/tuning/tmp.1/moses.ini’: No such file or directory --- If I change this line in the config file from decoding-steps = "t0,g0,t1,g1:t2" to decoding-steps = "t0,g0,t1,g1" then the training ends without errors. I'll appreciate suggestions on how to solve that. Alex ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Error on lmplz
Hello, today I've pulled moses code and recompiled and some experiments (EMS) that were already working are failing on the LM training step with the following error: Executing: /opt/moses/bin/lmplz --text /home/alexmc/devel/toydata/process/lm/nc=pos.factored.1 --order 5 --arpa /home/alexmc/devel/toydata/process/lm/nc=pos.lm.1 --discount_fallback === 1/5 Counting and sorting n-grams === Reading /mnt/a62/devel/toydata/process/lm/nc=pos.factored.1 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 tcmalloc: large alloc 4753956864 bytes == 0x1f7c000 @ tcmalloc: large alloc 22185107456 bytes == 0x11d536000 @ Unigram tokens 2433135 types 47 === 2/5 Calculating and sorting adjusted counts === Chain sizes: 1:564 2:2630656000 3:493248 4:7891967488 5:1150912 tcmalloc: large alloc 11509121024 bytes == 0x1f7c000 @ tcmalloc: large alloc 2630656000 bytes == 0x2aff7 @ tcmalloc: large alloc 4932485120 bytes == 0x34cc3a000 @ tcmalloc: large alloc 7891968000 bytes == 0x64933c000 @ lmplz: ./util/fixed_array.hh:104: T& util::FixedArray::operator[](std::size_t) [with T = lm::NGramStream; std::size_t = long unsigned int]: Assertion `i < size()' failed. I'm runing a Linux server with Ubuntu 15.04 Any help will be appreciated Alex Martínez___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] EMS Factored Model generation table
I'm getting this error on a EMS training during tuning phase, not sure if this configuration has sense, just trying to play with a POS tagger and with factored models: Executing: /opt/moses/bin/moses -v 0 -config /mnt/a62/devel/en_es/tuning/moses.filtered.ini.1 -weight-overwrite 'GenerationModel0= 0.038462 LM1= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 WordPenalty0= -0.128205 LM2= 0.064103 Distortion0= 0.038462 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM0= 0.064103 GenerationModel1= 0.038462 0.00 PhrasePenalty0= 0.025641' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out Can't read /mnt/a62/devel/en_es/model/generation.1-0 Exit code: 1 The decoder died. CONFIG WAS -weight-overwrite 'GenerationModel0= 0.038462 LM1= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 WordPenalty0= -0.128205 LM2= 0.064103 Distortion0= 0.038462 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 LM0= 0.064103 GenerationModel1= 0.038462 0.00 PhrasePenalty0= 0.025641' cp: cannot stat ‘/mnt/a62/devel/en_es/tuning/tmp.1/moses.ini’: No such file or directory I'm trying to use a factored model from en to es with the following factors: input-factors = pos lemma surface output-factors = pos lemma surface alignment-factors = "surface -> surface" translation-factors = "surface -> surface, pos -> pos" reordering-factors = "surface -> surface" generation-factors = "surface -> pos" decoding-steps = "t0,t1,g0" The corpus has been previously cleaned and factored with the format: pos0|lemma0|surface0 pos1|lemma1|surface1 pos2|lemma2|surface2 ... posN|lemmaN|surfaceN before factoring I've filtered the corpus to filter characters like | < > [] and also filtered the segments with length > 80 ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] unformatted biconcor output
Hi,i have succeeded installing the Web interface to the EMS and I'm playing with the config.toy. I still have a small problem... when I click over a source word to get the concordance (biconcor) the output I get is unformatted likefinally(15) enfin , ceux qui s' opposent à une réglementation sur les rémunérations seront sûrement ceux qui nous mettront en garde contre les « conséquences fortuites » . ||| finally , those opposing pay regulation are certain to warn us about " unintended consequences . " ||| 0 0 ||| 0 0 ||| 0-0 1-1 2-2 19-3 10-4 11-4 7-5 8-5 5-6 5-7 6-8 3-9 4-9 5-9 17-10 9-11 22-12 24-13 23-14 26-15 enfin , d' autres politiques des pays riches pourraient être plus efficaces que l' aide internationale . ||| finally , other rich country policies may be much more beneficial than aid . ||| 0 0 ||| 0 0 ||| 0-0 1-1 2-2 3-2 5-3 7-3 6-4 4-5 8-6 9-7 11-8 10-9 11-10 12-11 13-12 14-12 15-12 enfin , dans le cadre de l' appréciation de sociétés contrôlées , il importe de ne pas juger l' indépendance des administrateurs essentiellement sur leur indépendance vis @-@ à-vis du conseil de la société où ils siègent , mais plutôt vis @-@ à-vis de son actionnaire majoritaire . ||| finally , when assessing controlled companies , the independence of directors should not be judged largely by looking at the extent to which they are independent of the company on whose board they serve . rather , considerable attention should be given to their independence from the controlling shareholder . ||| 0 0 ||| 0 0 ||| 0-0 1-1 2-2 7-3 10-4 9-5 11-6 18-7 18-8 19-8 20-9 21-10 13-11 15-12 16-12 17-13 17-14 17-15 17-16 17-17 30-18 29-19 28-20 31-21 32-27 33-28 23-29 34-31 35-32 36-33 38-34 39-35 37-36 38-37 42-38 43-42 24-43 25-44 26-45 27-46 27-47 44-47 45-48 46-48 I have moved biconcor and the web directory to the root of my apache web server, have I missed something?Thanks in advanceAlex.___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
