how can i run from the grammar step?
--
Best Regards
Arezoo Arjomandzadeh
MSc student in Artificial Intelligence
Computer & IT engineering
Shahrood University of Technology, Iran
On Monday, August 14, 2017 4:58 AM, Matt Post <p...@cs.jhu.edu> wrote:
It looks like grammar sorting is failing. Check the logs to see why. Delete
grammar.* and try again from that step.
On Aug 14, 2017, at 10:49 AM, Arezoo Arjomand <arezooarjom...@yahoo.com> wrote:
Hi,I add "--alignergiza" to terminal command. It seems the alignment error is
fixed but the grammer error is still remain both for berkeley aligner and
giza. grammar.gz is empty and the runing dir is attached.
On Monday, August 14, 2017 2:08 AM, Matt Post <p...@cs.jhu.edu> wrote:
It looks like alignment failed. Is there a file alignments/training.align?
That is build from the two pieces, under alignments/0/giza.SRC-TRG (and
TRG-SRC) that failed.
On Aug 13, 2017, at 7:21 PM, Arezoo Arjomand <arezooarjom...@yahoo.com> wrote:
Hi,When I run the pipleline the following error is shown. The previous error ,
write in the previous email, is shown when i run the same dir for second time
and grammar.gz is empty.
how can i fix the folloewing error?
[source-numlines] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es [CHANGED]
cmd=cat /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es | wc -l
took 0 seconds (0s)
[source-numlines] retrieved cached result => 77457
[giza-0] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.es
[CHANGED]
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.en
[CHANGED]
dep=alignments/0/model/aligned.grow-diag-final [NOT FOUND]
cmd=rm -f alignments/0/corpus.0-0.*;
/home/arezoo1/joshua-tutorial/joshua/scripts/training/run-giza.pl --root-dir
alignments/0 -e en -f es -corpus
/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus -merge
grow-diag-final > alignments/0/giza.log 2>&1
*** Error in `/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal': double
free or corruption (out): 0x0000556a69b42160 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f91d0fb908b]
/lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f91d0fc26fa]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f91d0fc612c]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x2b5a)[0x556a6993ab5a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f91d0f603f1]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x5f4a)[0x556a6993df4a]
======= Memory map: ========
556a69938000-556a69941000 r-xp 00000000 08:0a 1051501
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b41000-556a69b42000 r--p 00009000 08:0a 1051501
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b42000-556a69b43000 rw-p 0000a000 08:0a 1051501
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b43000-556a69b45000 rw-p 00000000 00:00 0
556a6af09000-556a6afbf000 rw-p 00000000 00:00 0 [heap]
7f91cc000000-7f91cc021000 rw-p 00000000 00:00 0
7f91cc021000-7f91d0000000 ---p 00000000 00:00 0
7f91d0c37000-7f91d0d3f000 r-xp 00000000 08:0a 1708999
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0d3f000-7f91d0f3e000 ---p 00108000 08:0a 1708999
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3e000-7f91d0f3f000 r--p 00107000 08:0a 1708999
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3f000-7f91d0f40000 rw-p 00108000 08:0a 1708999
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f40000-7f91d10fd000 r-xp 00000000 08:0a 1708931
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d10fd000-7f91d12fd000 ---p 001bd000 08:0a 1708931
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d12fd000-7f91d1301000 r--p 001bd000 08:0a 1708931
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1301000-7f91d1303000 rw-p 001c1000 08:0a 1708931
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1303000-7f91d1307000 rw-p 00000000 00:00 0
7f91d1307000-7f91d131d000 r-xp 00000000 08:0a 1708971
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d131d000-7f91d151c000 ---p 00016000 08:0a 1708971
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151c000-7f91d151d000 r--p 00015000 08:0a 1708971
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151d000-7f91d151e000 rw-p 00016000 08:0a 1708971
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151e000-7f91d1697000 r-xp 00000000 08:0a 1976366
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1697000-7f91d1896000 ---p 00179000 08:0a 1976366
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1896000-7f91d18a0000 r--p 00178000 08:0a 1976366
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d18a0000-7f91d18a2000 rw-p 00182000 08:0a 1976366
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d18a2000-7f91d18a6000 rw-p 00000000 00:00 0
7f91d18a6000-7f91d18cb000 r-xp 00000000 08:0a 1708903
/lib/x86_64-linux-gnu/ld-2.24.so
7f91d1aa6000-7f91d1aaa000 rw-p 00000000 00:00 0
7f91d1ac7000-7f91d1acb000 rw-p 00000000 00:00 0
7f91d1acb000-7f91d1acc000 r--p 00025000 08:0a 1708903
/lib/x86_64-linux-gnu/ld-2.24.so
7f91d1acc000-7f91d1acd000 rw-p 00026000 08:0a 1708903
/lib/x86_64-linux-gnu/ld-2.24.so
7f91d1acd000-7f91d1ace000 rw-p 00000000 00:00 0
7ffc675c9000-7ffc675ea000 rw-p 00000000 00:00 0 [stack]
7ffc675f4000-7ffc675f6000 r--p 00000000 00:00 0 [vvar]
7ffc675f6000-7ffc675f8000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
JOB FAILED (return code 2)
[aligner-combine] rebuilding...
dep=alignments/0/model/aligned.grow-diag-final [CHANGED]
dep=alignments/training.align [NOT FOUND]
cmd=cat alignments/0/model/aligned.grow-diag-final > alignments/training.align
took 0 seconds (0s)
[thrax-input-file] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es [CHANGED]
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.en [CHANGED]
dep=alignments/training.align [CHANGED]
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file [NOT
FOUND]
cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/training/paste
/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es
/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.en
alignments/training.align | perl -pe 's/\t/ ||| /g' | grep -v '()' | grep -v
'||| \+$' > /home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file
took 1 seconds (1s)
[thrax-prep] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file
[CHANGED]
dep=grammar.gz [NOT FOUND]
cmd=hadoop fs -rm -r
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02; hadoop fs -mkdir
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02; hadoop fs -put
/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02/input-file
took 4 seconds (4s)
[thrax-run] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file
[CHANGED]
dep=thrax-phrase.conf [CHANGED]
dep=grammar.gz [NOT FOUND]
cmd=hadoop jar /home/arezoo1/joshua-tutorial/joshua/thrax/bin/thrax.jar -D
mapreduce.task.timeout=0 -D mapreduce.map.java.opts='-Xmx4g' -D
mapreduce.reduce.java.opts='-Xmx4g' -D hadoop.tmp.dir=/tmp thrax-phrase.conf
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02 > thrax.log 2>&1;
rm -f grammar grammar.gz; hadoop fs -cat
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02/final/* | gzip -cd
| /home/arezoo1/joshua-tutorial/joshua/scripts/training/filter-rules.pl -t 100
| gzip -9n > grammar.gz
took 28 seconds (28s)
17/08/13 13:16:06 INFO Configuration.deprecation: io.bytes.per.checksum is
deprecated. Instead, use dfs.bytes-per-checksum
17/08/13 13:16:06 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02
[pack-grammar] rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/vocabulary [NOT
FOUND]
dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/encoding [NOT FOUND]
dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/slice_00000.source
[NOT FOUND]
cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/support/grammar-packer.pl -a
-T /tmp -m 8g -g grammar.gz -o
/home/arezoo1/joshua-tutorial/runs/02/grammar.packed
JOB FAILED (return code 1)
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gzAte7) to current directory.
On Saturday, August 12, 2017 2:03 AM, Matt Post <p...@cs.jhu.edu> wrote:
You probably have an empty grammar. What's the file size of grammar.gz?
On Aug 8, 2017, at 12:40 PM, Arezoo Arjomand <arezooarjom...@yahoo.com> wrote:
Hi,
When I run the pipeline the following error was accured. how can i fix it?
Deleted
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_2[pack-grammar]
rebuilding...
dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/vocabulary [NOT FOUND]
dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/encoding [NOT FOUND]
dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/slice_00000.source
[NOT FOUND]
cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/support/grammar-packer.pl -a
-T /tmp -m 8g -g grammar.gz -o
/home/arezoo1/joshua-tutorial/runs/2/grammar.packed
JOB FAILED (return code 1)
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gzegdu) to current directory.
<920.zip>
