subject:"\[jira\] \[Commented\] \(JOSHUA\-277\) UnsatisfiedLinkError\: no ken in java.library.path"

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-28 Thread Matt Post

Hi,

There is no Joshua manual, unfortunately, just the Confluence pages.

I looked at your run and it seems that Thrax is failing. I don't know what your 
Hadoop configuration is like, but that is likely the problem (see thrax.log in 
these directories). If you setup Hadoop incorrectly, or don't have enough 
space, or set it up on a network share instead of local disks, all of these 
things can cause problems.

matt


> On Aug 28, 2017, at 2:46 PM, Jeffrey Smith (JIRA)  wrote:
> 
> 
>[ 
> https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143731#comment-16143731
>  ] 
> 
> Jeffrey Smith commented on JOSHUA-277:
> --
> 
> Thank you for your help. Here are the two directories in question. Also, do
> you know of a manual, similar to the moses manual, for joshua? I couldn't
> seem to find one.
> Moses does work on this same computer I am running this on.
> 
> joshua-tutorial.tar.gz
> 
> 
> joshua.tar.gz
> 
> 
> 
> 
> 
> 
> 
> -- 
> *Jeffrey Smith, PhD*
> Chief Systems Engineer and E2 Lead
> Multi Agency Collaboration Environment (MACE)
> Sierra Nevada Corporation
> 3076 Centreville Road, Herndon, VA 20171
> 703-464-6434 (Office)
> 603-566-0124 (Cell)
> jeff.sm...@macefusion.com
> 
> 
>> UnsatisfiedLinkError: no ken in java.library.path
>> -
>> 
>>Key: JOSHUA-277
>>URL: https://issues.apache.org/jira/browse/JOSHUA-277
>>Project: Joshua
>> Issue Type: Bug
>>   Reporter: Thamme Gowda
>> 
>> I followed this guide 
>> http://joshua.incubator.apache.org/6.0/quick-start.html to test the latest 
>> build.
>> Assuming there few things are broken due to newer maven build system, I 
>> tried to fix pipeline.pl to get the quick start guide working.
>> Which files from kenlm build should I add to JNI path? (I am unable to 
>> locate the library file in the kenlm build output)
>> Here is the full log:
>> {code}
>> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
>> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en  
>>--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
>> [train-copy-and-filter] cached, skipping...
>> [train-vocab-bn] cached, skipping...
>> [train-vocab-en] cached, skipping...
>> [tune-copy-and-filter] cached, skipping...
>> [tune-vocab-bn] cached, skipping...
>> [tune-vocab-en.0] cached, skipping...
>> [tune-vocab-en.1] cached, skipping...
>> [tune-vocab-en.2] cached, skipping...
>> [tune-vocab-en.3] cached, skipping...
>> [test-copy-and-filter] cached, skipping...
>> [test-vocab-bn] cached, skipping...
>> [test-vocab-en.0] cached, skipping...
>> [test-vocab-en.1] cached, skipping...
>> [test-vocab-en.2] cached, skipping...
>> [test-vocab-en.3] cached, skipping...
>> [source-numlines] cached, skipping...
>> [source-numlines] retrieved cached result =>20788
>> [berkeley-aligner-chunk-0] cached, skipping...
>> [aligner-combine] cached, skipping...
>> [pack-grammar] cached, skipping...
>> [lm-sort-uniq] cached, skipping...
>> [kenlm] cached, skipping...
>> [compile-kenlm] cached, skipping...
>> [glue-tune] cached, skipping...
>> Error: Could not find or load main class 
>> joshua.util.encoding.EncoderConfiguration
>> [tune-bundle] cached, skipping...
>> [mert-1] rebuilding...
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  [CHANGED]
>>  dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>>  [NOT FOUND]
>>  
>> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>>  --tunedir 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune 
>> --tuner mert --decoder 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>>  --decoder-config 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  --decoder-output-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>>  --decoder-log-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>>  --iterations 10 --metric 'BLEU 4 closest'
>>  JOB FAILED (return code 1)
>> Exception in thread "main" java.lang.RuntimeException:

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-28 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143731#comment-16143731
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

Thank you for your help. Here are the two directories in question. Also, do
you know of a manual, similar to the moses manual, for joshua? I couldn't
seem to find one.
Moses does work on this same computer I am running this on.

 joshua-tutorial.tar.gz


 joshua.tar.gz







-- 
*Jeffrey Smith, PhD*
Chief Systems Engineer and E2 Lead
Multi Agency Collaboration Environment (MACE)
Sierra Nevada Corporation
3076 Centreville Road, Herndon, VA 20171
703-464-6434 (Office)
603-566-0124 (Cell)
jeff.sm...@macefusion.com


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-28 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143709#comment-16143709
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

That is so nice. I will tar up 2 directories. The joshua and joshua-tutorial 
directories. Thank you.

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-27 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143361#comment-16143361
 ] 

Matt Post commented on JOSHUA-277:
--

If you want to tar up your whole run directory and make it available somewhere 
I can take a closer look.





> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-27 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143284#comment-16143284
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

I deleted and reinstalled joshua. I didn't touch /data/joshua-tutorial 
directory. I still get same error. Not sure where to go from here? Any 
suggestions.


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
>

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-26 Thread Matt Post

You said you're on OS X? This should work, but you might try building in a 
Docker container. There's a Dockerfile in distribution/docker/kenlm


> On Aug 25, 2017, at 1:24 PM, Jeffrey Smith (JIRA)  wrote:
> 
> 
>[ 
> https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141502#comment-16141502
>  ] 
> 
> Jeffrey Smith commented on JOSHUA-277:
> --
> 
> Thanks. I appreciate you getting back to me on this. I only jave JDK 8 on 
> this system. I did the above steps with the same issue. Perhaps this is a 
> problem.
> 
> Joshua is here:
> [ec2-user@ip-172-31-4-253 runs]$ echo $JOSHUA
> /data/joshua
> 
> I installed the joshua tutorial files in:
> /data/joshua-tutorial so I am running tutorial from 
> /data/joshua-tutorial/runs
> 
> when I run:
> $JOSHUA/bin/pipeline.pl \
>  --rundir 1 \
>  --readme "Baseline Hiero run" \
>  --source es \
>  --target en \
>  --type hiero \
>  --corpus $FISHER/corpus/asr/fisher_train \
>  --tune $FISHER/corpus/asr/fisher_dev \
>  --test $FISHER/corpus/asr/fisher_dev2 \
>  --maxlen 11 \
>  --maxlen-tune 11 \
>  --maxlen-test 11 \
>  --tuner-iterations 1 \
>  --lm-order 3
> 
> I still get the error I described
> 
> 
> 
>> UnsatisfiedLinkError: no ken in java.library.path
>> -
>> 
>>Key: JOSHUA-277
>>URL: https://issues.apache.org/jira/browse/JOSHUA-277
>>Project: Joshua
>> Issue Type: Bug
>>   Reporter: Thamme Gowda
>> 
>> I followed this guide 
>> http://joshua.incubator.apache.org/6.0/quick-start.html to test the latest 
>> build.
>> Assuming there few things are broken due to newer maven build system, I 
>> tried to fix pipeline.pl to get the quick start guide working.
>> Which files from kenlm build should I add to JNI path? (I am unable to 
>> locate the library file in the kenlm build output)
>> Here is the full log:
>> {code}
>> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
>> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en  
>>--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
>> [train-copy-and-filter] cached, skipping...
>> [train-vocab-bn] cached, skipping...
>> [train-vocab-en] cached, skipping...
>> [tune-copy-and-filter] cached, skipping...
>> [tune-vocab-bn] cached, skipping...
>> [tune-vocab-en.0] cached, skipping...
>> [tune-vocab-en.1] cached, skipping...
>> [tune-vocab-en.2] cached, skipping...
>> [tune-vocab-en.3] cached, skipping...
>> [test-copy-and-filter] cached, skipping...
>> [test-vocab-bn] cached, skipping...
>> [test-vocab-en.0] cached, skipping...
>> [test-vocab-en.1] cached, skipping...
>> [test-vocab-en.2] cached, skipping...
>> [test-vocab-en.3] cached, skipping...
>> [source-numlines] cached, skipping...
>> [source-numlines] retrieved cached result =>20788
>> [berkeley-aligner-chunk-0] cached, skipping...
>> [aligner-combine] cached, skipping...
>> [pack-grammar] cached, skipping...
>> [lm-sort-uniq] cached, skipping...
>> [kenlm] cached, skipping...
>> [compile-kenlm] cached, skipping...
>> [glue-tune] cached, skipping...
>> Error: Could not find or load main class 
>> joshua.util.encoding.EncoderConfiguration
>> [tune-bundle] cached, skipping...
>> [mert-1] rebuilding...
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  [CHANGED]
>>  dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>>  [NOT FOUND]
>>  
>> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>>  --tunedir 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune 
>> --tuner mert --decoder 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>>  --decoder-config 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  --decoder-output-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>>  --decoder-log-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>>  --iterations 10 --metric 'BLEU 4 closest'
>>  JOB FAILED (return code 1)
>> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
>> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>>  at 
>>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-25 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141502#comment-16141502
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

Thanks. I appreciate you getting back to me on this. I only jave JDK 8 on this 
system. I did the above steps with the same issue. Perhaps this is a problem.

Joshua is here:
[ec2-user@ip-172-31-4-253 runs]$ echo $JOSHUA
/data/joshua

I installed the joshua tutorial files in:
/data/joshua-tutorial so I am running tutorial from 
/data/joshua-tutorial/runs

when I run:
$JOSHUA/bin/pipeline.pl \
  --rundir 1 \
  --readme "Baseline Hiero run" \
  --source es \
  --target en \
  --type hiero \
  --corpus $FISHER/corpus/asr/fisher_train \
  --tune $FISHER/corpus/asr/fisher_dev \
  --test $FISHER/corpus/asr/fisher_dev2 \
  --maxlen 11 \
  --maxlen-tune 11 \
  --maxlen-test 11 \
  --tuner-iterations 1 \
  --lm-order 3

I still get the error I described



> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-24 Thread Thamme Gowda (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140845#comment-16140845
 ] 

Thamme Gowda commented on JOSHUA-277:
-

> Exception in thread "main" java.util.NoSuchElementException at 
> org.apache.joshua.util.io.LineReader.next(LineReader.java:276)

this usually happens when you are dealing with version of classes, libraries, 
or JDK.
Suggestion:
verify that you have JDK 8 active in your environment
do a fresh build of joshua:
 - go to master branch: git checkout master.
 - get latest changes: git pull origin master
 - Rebuild: mvn clean package
 - export JOSHUA= correct directory, $PWD where you cloned Joshua

and repeat your pipeline.



> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
>

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-23 Thread Matt Post

What's the file size of grammar.gz? Looks like it didn't get extracted.


> On Aug 23, 2017, at 8:14 PM, Jeffrey Smith (JIRA)  wrote:
> 
> 
>[ 
> https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138814#comment-16138814
>  ] 
> 
> Jeffrey Smith commented on JOSHUA-277:
> --
> 
> Getting closer:
> 
> I deleted runs/1 and re-ran:
> 
> $JOSHUA/bin/pipeline.pl \
> --rundir 1 \
> --readme "Baseline Hiero run" \
> --source es \
> --target en \
> --type hiero \
> --corpus $FISHER/corpus/asr/fisher_train \
> --tune $FISHER/corpus/asr/fisher_dev \
> --test $FISHER/corpus/asr/fisher_dev2 \
> --maxlen 11 \
> --maxlen-tune 11 \
> --maxlen-test 11 \
> --tuner-iterations 1 \
> --lm-order 3
> 
> The example got farther but ended with the following error. 
> ...
> * Packing grammar at "grammar.gz" to 
> "/data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
> * Running the grammar-packer.pl script with the command: 
> /data/joshua/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o 
> /data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed
> Exception in thread "main" java.util.NoSuchElementException
>at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>at 
> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>at 
> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>at 
> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
> * FATAL: Couldn't pack the grammar.
> 
> 
>> UnsatisfiedLinkError: no ken in java.library.path
>> -
>> 
>>Key: JOSHUA-277
>>URL: https://issues.apache.org/jira/browse/JOSHUA-277
>>Project: Joshua
>> Issue Type: Bug
>>   Reporter: Thamme Gowda
>> 
>> I followed this guide 
>> http://joshua.incubator.apache.org/6.0/quick-start.html to test the latest 
>> build.
>> Assuming there few things are broken due to newer maven build system, I 
>> tried to fix pipeline.pl to get the quick start guide working.
>> Which files from kenlm build should I add to JNI path? (I am unable to 
>> locate the library file in the kenlm build output)
>> Here is the full log:
>> {code}
>> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
>> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en  
>>--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
>> [train-copy-and-filter] cached, skipping...
>> [train-vocab-bn] cached, skipping...
>> [train-vocab-en] cached, skipping...
>> [tune-copy-and-filter] cached, skipping...
>> [tune-vocab-bn] cached, skipping...
>> [tune-vocab-en.0] cached, skipping...
>> [tune-vocab-en.1] cached, skipping...
>> [tune-vocab-en.2] cached, skipping...
>> [tune-vocab-en.3] cached, skipping...
>> [test-copy-and-filter] cached, skipping...
>> [test-vocab-bn] cached, skipping...
>> [test-vocab-en.0] cached, skipping...
>> [test-vocab-en.1] cached, skipping...
>> [test-vocab-en.2] cached, skipping...
>> [test-vocab-en.3] cached, skipping...
>> [source-numlines] cached, skipping...
>> [source-numlines] retrieved cached result =>20788
>> [berkeley-aligner-chunk-0] cached, skipping...
>> [aligner-combine] cached, skipping...
>> [pack-grammar] cached, skipping...
>> [lm-sort-uniq] cached, skipping...
>> [kenlm] cached, skipping...
>> [compile-kenlm] cached, skipping...
>> [glue-tune] cached, skipping...
>> Error: Could not find or load main class 
>> joshua.util.encoding.EncoderConfiguration
>> [tune-bundle] cached, skipping...
>> [mert-1] rebuilding...
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  [CHANGED]
>>  dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>>  [NOT FOUND]
>>  
>> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>>  --tunedir 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune 
>> --tuner mert --decoder 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>>  --decoder-config 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  --decoder-output-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>>  --decoder-log-file 
>>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-23 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138814#comment-16138814
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

Getting closer:

I deleted runs/1 and re-ran:

$JOSHUA/bin/pipeline.pl \
--rundir 1 \
--readme "Baseline Hiero run" \
--source es \
--target en \
--type hiero \
--corpus $FISHER/corpus/asr/fisher_train \
--tune $FISHER/corpus/asr/fisher_dev \
--test $FISHER/corpus/asr/fisher_dev2 \
--maxlen 11 \
--maxlen-tune 11 \
--maxlen-test 11 \
--tuner-iterations 1 \
--lm-order 3

The example got farther but ended with the following error. 
...
* Packing grammar at "grammar.gz" to 
"/data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
* Running the grammar-packer.pl script with the command: 
/data/joshua/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o 
/data/joshua-tutorial/runs/1/tune/model/grammar.gz.packed
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at 
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at 
org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at 
org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
>

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-23 Thread Matt Post

what is the file size of lm dot kenlm and lm.gz? that will tell you if they 
built fine. 

check that joshua config path to lm is valid. thrown error might be off. 

matt (from my phone)

> Le 23 août 2017 à 15:27, Jeffrey Smith (JIRA)  a écrit :
> 
> 
>[ 
> https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138334#comment-16138334
>  ] 
> 
> Jeffrey Smith commented on JOSHUA-277:
> --
> 
> PS. There is a  "runs/1/tune/model/lm.kenlm". It is a soft-link to 
> .../joshua-tutorial/runs/1/lm.kenlm . Perhaps this is not what it is supposed 
> to be?
> 
> 
>> UnsatisfiedLinkError: no ken in java.library.path
>> -
>> 
>>Key: JOSHUA-277
>>URL: https://issues.apache.org/jira/browse/JOSHUA-277
>>Project: Joshua
>> Issue Type: Bug
>>   Reporter: Thamme Gowda
>> 
>> I followed this guide 
>> http://joshua.incubator.apache.org/6.0/quick-start.html to test the latest 
>> build.
>> Assuming there few things are broken due to newer maven build system, I 
>> tried to fix pipeline.pl to get the quick start guide working.
>> Which files from kenlm build should I add to JNI path? (I am unable to 
>> locate the library file in the kenlm build output)
>> Here is the full log:
>> {code}
>> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
>> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en  
>>--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
>> [train-copy-and-filter] cached, skipping...
>> [train-vocab-bn] cached, skipping...
>> [train-vocab-en] cached, skipping...
>> [tune-copy-and-filter] cached, skipping...
>> [tune-vocab-bn] cached, skipping...
>> [tune-vocab-en.0] cached, skipping...
>> [tune-vocab-en.1] cached, skipping...
>> [tune-vocab-en.2] cached, skipping...
>> [tune-vocab-en.3] cached, skipping...
>> [test-copy-and-filter] cached, skipping...
>> [test-vocab-bn] cached, skipping...
>> [test-vocab-en.0] cached, skipping...
>> [test-vocab-en.1] cached, skipping...
>> [test-vocab-en.2] cached, skipping...
>> [test-vocab-en.3] cached, skipping...
>> [source-numlines] cached, skipping...
>> [source-numlines] retrieved cached result =>20788
>> [berkeley-aligner-chunk-0] cached, skipping...
>> [aligner-combine] cached, skipping...
>> [pack-grammar] cached, skipping...
>> [lm-sort-uniq] cached, skipping...
>> [kenlm] cached, skipping...
>> [compile-kenlm] cached, skipping...
>> [glue-tune] cached, skipping...
>> Error: Could not find or load main class 
>> joshua.util.encoding.EncoderConfiguration
>> [tune-bundle] cached, skipping...
>> [mert-1] rebuilding...
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  [CHANGED]
>>  dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>>  
>> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>>  [NOT FOUND]
>>  
>> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>>  
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>>  --tunedir 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune 
>> --tuner mert --decoder 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>>  --decoder-config 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>>  --decoder-output-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>>  --decoder-log-file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>>  --iterations 10 --metric 'BLEU 4 closest'
>>  JOB FAILED (return code 1)
>> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
>> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
>> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>>at 
>> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>>at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>>at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>>at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
>> Caused by: java.lang.reflect.InvocationTargetException
>>at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>at 
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>at 
>>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-23 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138334#comment-16138334
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

PS. There is a  "runs/1/tune/model/lm.kenlm". It is a soft-link to 
.../joshua-tutorial/runs/1/lm.kenlm . Perhaps this is not what it is supposed 
to be?


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-23 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138329#comment-16138329
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

Good point. I should have mentioned this. I am using Red Hat Enterprise Linux 
7.4 on an xlarge Amazon EC2 instance. Thanks for the hint. I will investigate.

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Thamme Gowda (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137728#comment-16137728
 ] 

Thamme Gowda commented on JOSHUA-277:
-

 I (the guy who originally created this issue) had faced this problem while 
trying to make Joshua run on OSX.

I can confirm that Joshua pipeline works without any hurdles on Linux. I had 
previously built it on Ubuntu and a few days back I had built Joshua and 
trained few models on Centos. it worked without any trouble. (I am not sure on 
OSX yet)

[~jesmitty] Which is your OS? 
Most likely, one of the dependencies (Kenlm) is not built or picked up by 
joshua.

Hint: debug why "runs/1/tune/model/lm.kenlm" is not being created when you run 
the pipeline.

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137681#comment-16137681
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

There is a there a file lib/libken.s and bin/lmplz after I  ran 
download-deps.sh. Perhaps I should give whole output. It is below. it is in 
response to the first part of the tutorial (get similar results with other 
parts) - $JOSHUA/bin/pipeline.pl \
  --rundir 1 \
  --readme "Baseline Hiero run" \
  --source es \
  --target en \
  --type hiero \
  --corpus $FISHER/corpus/asr/fisher_train \
  --tune $FISHER/corpus/asr/fisher_dev \
  --test $FISHER/corpus/asr/fisher_dev2 \
  --maxlen 11 \
  --maxlen-tune 11 \
  --maxlen-test 11 \
  --tuner-iterations 1 \
  --lm-order 3
_

source test1.sh
[train-copy-and-filter] cached, skipping...
[train-tokenize-es] cached, skipping...
[train-tokenize-en] cached, skipping...
[train-trim] cached, skipping...
[train-lowercase-es] cached, skipping...
[train-lowercase-en] cached, skipping...
[train-vocab-es] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-tokenize-es] cached, skipping...
[tune-tokenize-en.0] cached, skipping...
[tune-tokenize-en.1] cached, skipping...
[tune-tokenize-en.2] cached, skipping...
[tune-tokenize-en.3] cached, skipping...
[tune-trim] cached, skipping...
[tune-lowercase-es] cached, skipping...
[tune-lowercase-en.0] cached, skipping...
[tune-lowercase-en.1] cached, skipping...
[tune-lowercase-en.2] cached, skipping...
[tune-lowercase-en.3] cached, skipping...
[tune-vocab-es] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-tokenize-es] cached, skipping...
[test-tokenize-en.0] cached, skipping...
[test-tokenize-en.1] cached, skipping...
[test-tokenize-en.2] cached, skipping...
[test-tokenize-en.3] cached, skipping...
[test-trim] cached, skipping...
[test-lowercase-es] cached, skipping...
[test-lowercase-en.0] cached, skipping...
[test-lowercase-en.1] cached, skipping...
[test-lowercase-en.2] cached, skipping...
[test-lowercase-en.3] cached, skipping...
[test-vocab-es] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
  dep=/data/joshua-tutorial/runs/1/data/tune/corpus.es [CHANGED]
  dep=/data/joshua-tutorial/runs/1/tune/joshua.config [CHANGED]
  dep=tune/model/grammar.gz.packed/slice_0.source [CHANGED]
  dep=/data/joshua-tutorial/runs/1/tune/joshua.config.final [NOT FOUND]
  cmd=/data/joshua/scripts/training/run_tuner.py 
/data/joshua-tutorial/runs/1/data/tune/corpus.es 
/data/joshua-tutorial/runs/1/data/tune/corpus.en --tunedir 
/data/joshua-tutorial/runs/1/tune --tuner mert --decoder 
/data/joshua-tutorial/runs/1/tune/decoder_command --decoder-config 
/data/joshua-tutorial/runs/1/tune/joshua.config --decoder-output-file 
/data/joshua-tutorial/runs/1/tune/output.nbest --decoder-log-file 
/data/joshua-tutorial/runs/1/tune/joshua.log --iterations 1 --metric 'BLEU 4 
closest'
  JOB FAILED (return code 1)
/data/joshua/ext/kenlm/util/file.cc:76 in int util::OpenReadOrThrow(const 
char*) threw ErrnoException because `-1 == (ret = open(name, 00))'.
No such file or directory while opening 
/home/ec2-user/data/joshua-tutorial/runs/1/tune/model/lm.kenlm
Traceback (most recent call last):
  File "/data/joshua/scripts/training/run_tuner.py", line 554, in 
main(sys.argv)
  File "/data/joshua/scripts/training/run_tuner.py", line 537, in main
run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, 
opts.decoder_config, opts.decoder_output_file, opts)
  File "/data/joshua/scripts/training/run_tuner.py", line 418, in run_zmert
opts.metric, opts.iterations or 10)
  File "/data/joshua/scripts/training/run_tuner.py", line 400, in setup_configs
for feature,weight in get_features(config):
  File "/data/joshua/scripts/training/run_tuner.py", line 351, in get_features
output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % 
(JOSHUA, config_file), shell=True)
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '/data/joshua/bin/joshua-decoder -c 
/data/joshua-tutorial/runs/1/tune/joshua.config -show-weights -v 0' returned 
non-zero exit status -6


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL:

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137288#comment-16137288
 ] 

Matt Post commented on JOSHUA-277:
--

Is there a file lib/libken.so? And bin/lmplz?

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137279#comment-16137279
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

KenLM did build. I did start on Getting Started and ran download-deps.sh

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137271#comment-16137271
 ] 

Matt Post commented on JOSHUA-277:
--

KenLM is not getting built. Did you check out the Getting Started page? 
download-deps.sh downloads and builds KenLM. Likely it is failing.

https://cwiki.apache.org/confluence/display/JOSHUA/Getting+Started

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136879#comment-16136879
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

I seems like, in this example, the problem started at the message: 
"dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
 [NOT FOUND]". I have the same issue and this issue leads to the same error 
messages. Why would joshua.config.final not be found?

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by:

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136869#comment-16136869
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

I don't think it is a good idea to close this ticket.

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2017-08-22 Thread Jeffrey Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136867#comment-16136867
 ] 

Jeffrey Smith commented on JOSHUA-277:
--

Somehow the updated tutorial was supposed to have solved this problem. I have 
exactly the same problem too, even after going through the new tutorial link. 
Which step was supposed to have fixed the problem?

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2016-06-19 Thread Thamme Gowda (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338604#comment-15338604
 ] 

Thamme Gowda commented on JOSHUA-277:
-

Thanks for the link, [~post]. I will follow this tutorial.


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2016-06-18 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338337#comment-15338337
 ] 

Matt Post commented on JOSHUA-277:
--

Hi Thamme — I'm not sure if you want to try to go through this again, but I've 
updated the tutorial for the current state of Joshua: 
https://cwiki.apache.org/confluence/display/JOSHUA/Joshua+Tutorial. That would 
be more relevant than the one you tried to go through.




> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

Re: [jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

23 matches

Site Navigation

Mail list logo

Footer information