What's the file size of grammar.gz? Looks like it didn't get extracted.
> On Aug 23, 2017, at 8:14 PM, Jeffrey Smith (JIRA) <j...@apache.org> wrote: > > > [ > https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138814#comment-16138814 > ] > > Jeffrey Smith commented on JOSHUA-277: > -------------------------------------- > > Getting closer: > > I deleted runs/1 and re-ran: > > $JOSHUA/bin/pipeline.pl \ > --rundir 1 \ > --readme "Baseline Hiero run" \ > --source es \ > --target en \ > --type hiero \ > --corpus $FISHER/corpus/asr/fisher_train \ > --tune $FISHER/corpus/asr/fisher_dev \ > --test $FISHER/corpus/asr/fisher_dev2 \ > --maxlen 11 \ > --maxlen-tune 11 \ > --maxlen-test 11 \ > --tuner-iterations 1 \ > --lm-order 3 > > The example got farther but ended with the following error. > ... > * Packing grammar at "grammar.gz" to > "/data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed" > * Running the grammar-packer.pl script with the command: > /data/joshua/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o > /data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed > Exception in thread "main" java.util.NoSuchElementException > at org.apache.joshua.util.io.LineReader.next(LineReader.java:276) > at > org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239) > at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184) > at > org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120) > at > org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137) > * FATAL: Couldn't pack the grammar. > > >> UnsatisfiedLinkError: no ken in java.library.path >> ------------------------------------------------- >> >> Key: JOSHUA-277 >> URL: https://issues.apache.org/jira/browse/JOSHUA-277 >> Project: Joshua >> Issue Type: Bug >> Reporter: Thamme Gowda >> >> I followed this guide >> http://joshua.incubator.apache.org/6.0/quick-start.html to test the latest >> build. >> Assuming there few things are broken due to newer maven build system, I >> tried to fix pipeline.pl to get the quick start guide working. >> Which files from kenlm build should I add to JNI path? (I am unable to >> locate the library file in the kenlm build output) >> Here is the full log: >> {code} >> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero >> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en >> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en >> [train-copy-and-filter] cached, skipping... >> [train-vocab-bn] cached, skipping... >> [train-vocab-en] cached, skipping... >> [tune-copy-and-filter] cached, skipping... >> [tune-vocab-bn] cached, skipping... >> [tune-vocab-en.0] cached, skipping... >> [tune-vocab-en.1] cached, skipping... >> [tune-vocab-en.2] cached, skipping... >> [tune-vocab-en.3] cached, skipping... >> [test-copy-and-filter] cached, skipping... >> [test-vocab-bn] cached, skipping... >> [test-vocab-en.0] cached, skipping... >> [test-vocab-en.1] cached, skipping... >> [test-vocab-en.2] cached, skipping... >> [test-vocab-en.3] cached, skipping... >> [source-numlines] cached, skipping... >> [source-numlines] retrieved cached result => 20788 >> [berkeley-aligner-chunk-0] cached, skipping... >> [aligner-combine] cached, skipping... >> [pack-grammar] cached, skipping... >> [lm-sort-uniq] cached, skipping... >> [kenlm] cached, skipping... >> [compile-kenlm] cached, skipping... >> [glue-tune] cached, skipping... >> Error: Could not find or load main class >> joshua.util.encoding.EncoderConfiguration >> [tune-bundle] cached, skipping... >> [mert-1] rebuilding... >> >> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn >> [CHANGED] >> >> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config >> [CHANGED] >> dep=tune/model/grammar.packed/slice_00000.source [CHANGED] >> >> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final >> [NOT FOUND] >> >> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py >> >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn >> >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en >> --tunedir >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune >> --tuner mert --decoder >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command >> --decoder-config >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config >> --decoder-output-file >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest >> --decoder-log-file >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log >> --iterations 10 --metric 'BLEU 4 closest' >> JOB FAILED (return code 1) >> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate >> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'! >> at >> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761) >> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514) >> at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:122) >> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69) >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> at >> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757) >> ... 3 more >> Caused by: java.lang.ExceptionInInitializerError >> at >> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75) >> at >> org.apache.joshua.decoder.ff.lm.LanguageModelFF.<init>(LanguageModelFF.java:156) >> at >> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.<init>(StateMinimizingLanguageModel.java:50) >> ... 8 more >> Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no >> ken in java.library.path >> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:52) >> ... 11 more >> Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path >> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) >> at java.lang.Runtime.loadLibrary0(Runtime.java:870) >> at java.lang.System.loadLibrary(System.java:1122) >> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:43) >> ... 11 more >> Traceback (most recent call last): >> File >> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", >> line 553, in <module> >> main(sys.argv) >> File >> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", >> line 536, in main >> run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, >> opts.decoder_config, opts.decoder_output_file, opts) >> File >> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", >> line 417, in run_zmert >> opts.metric, opts.iterations or 10) >> File >> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", >> line 399, in setup_configs >> for feature,weight in get_features(config): >> File >> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", >> line 351, in get_features >> output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % >> (JOSHUA, config_file), shell=True) >> File >> "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", >> line 573, in check_output >> raise CalledProcessError(retcode, cmd, output=output) >> subprocess.CalledProcessError: Command >> '/Users/thammegr/work/projects/apache/incubator-joshua/bin/joshua-decoder -c >> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config >> -show-weights -v 0' returned non-zero exit status 1 >> {code} > > > > -- > This message was sent by Atlassian JIRA > (v6.4.14#64029)
