[GitHub] incubator-joshua pull request #:

2016-06-19 Thread fhieber 
Github user fhieber commented on the pull request:


https://github.com/apache/incubator-joshua/commit/6d2213a20b74432fc7cb131c732f7507b74053e9#commitcomment-17928740
  
In src/main/java/org/apache/joshua/decoder/StructuredTranslation.java:
In src/main/java/org/apache/joshua/decoder/StructuredTranslation.java on 
line 57:
Hi matt,

Do you need the features member of structuredTranslation to be a 
FeatureVector? The problem is that this leaks IDs of features into downstream 
applications. 

Cheers,
Felix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: joshua_master #63

2016-06-19 Thread Apache Jenkins Server 
See 

Changes:

[post] added removal of custom rules

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-5 (docker Ubuntu ubuntu5 ubuntu yahoo-not-h2) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/incubator-joshua.git # timeout=10
Fetching upstream changes from 
https://git-wip-us.apache.org/repos/asf/incubator-joshua.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/incubator-joshua.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision da124c2002d44b67485bd5eb138482b56bbc47ef 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f da124c2002d44b67485bd5eb138482b56bbc47ef
 > git rev-list 6d2213a20b74432fc7cb131c732f7507b74053e9 # timeout=10
[joshua_master] $ /home/jenkins/tools/maven/apache-maven-3.0.4/bin/mvn clean 
deploy javadoc:aggregate
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Apache Joshua Machine Translation Toolkit 6.0.6-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ joshua ---
[INFO] Deleting 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.2.1:process (default) @ joshua ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ joshua ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ joshua ---
[INFO] Compiling 264 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
joshua ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 319 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
joshua ---
[INFO] Compiling 41 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ joshua ---

---
 T E S T S
---
Running TestSuite
WARN - sentence 0 too long 401, truncating to length 200
WARN - sentence 0 too long 401, truncating to length 200
WARN - sentence 0 too long 401, truncating to length 200
WARN - sentence 0 too long 401, truncating to length 200
%
%
%
%
%
%
%
%
%
WARN - no grammars supplied!  Supplying dummy glue grammar.
WARN - no grammars supplied!  Supplying dummy glue grammar.
WARN - no grammars supplied!  Supplying dummy glue grammar.
WARN - no grammars supplied!  Supplying dummy glue grammar.
Tests run: 122, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 2.058 sec <<< 
FAILURE! - in TestSuite
basicVocabTest(org.apache.joshua.corpus.vocab.VocabularyTest)  Time elapsed: 
0.007 sec  <<< FAILURE!
java.lang.AssertionError: null
at 
org.apache.joshua.corpus.vocab.VocabularyTest.basicVocabTest(VocabularyTest.java:56)


Results :

Failed tests: 
  VocabularyTest.basicVocabTest:56 null

Tests run: 122, Failures: 1, Errors: 0, Skipped: 1

[INFO] 
[INFO] 
[INFO] Skipping Apache Joshua Machine Translation Toolkit
[INFO] This project has been banned from the build due to previous failures.
[INFO] 
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 14.480s
[INFO] Finished at: Mon Jun 20 03:14:40 UTC 2016
[INFO] Final Memory: 34M/641M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on 
project joshua: There are test failures.
[ERROR] 
[ERROR] Please refer to 
 for 
the individual

Build failed in Jenkins: joshua_master #62

2016-06-19 Thread Apache Jenkins Server 
See 

Changes:

[post] added partial translation string formatting to structured translation

--
[...truncated 1234 lines...]
1762/2297 KB   
1766/2297 KB   
1770/2297 KB   
1774/2297 KB   
1778/2297 KB   
1782/2297 KB   
1786/2297 KB   
1790/2297 KB   
1794/2297 KB   
1798/2297 KB   
1802/2297 KB   
1806/2297 KB   
1810/2297 KB   
1814/2297 KB   
1818/2297 KB   
1822/2297 KB   
1826/2297 KB   
1830/2297 KB   
1834/2297 KB   
1838/2297 KB   
1842/2297 KB   
1846/2297 KB   
1850/2297 KB   
1854/2297 KB   
1858/2297 KB   
1862/2297 KB   
1866/2297 KB   
1870/2297 KB   
1874/2297 KB   
1878/2297 KB   
1882/2297 KB   
1886/2297 KB   
1890/2297 KB   
1894/2297 KB   
1898/2297 KB   
1902/2297 KB   
1906/2297 KB   
1910/2297 KB   
1914/2297 KB   
1918/2297 KB   
1922/2297 KB   
1926/2297 KB   
1930/2297 KB   
1934/2297 KB   
1938/2297 KB   
1942/2297 KB   
1946/2297 KB   
1950/2297 KB   
1954/2297 KB   
1958/2297 KB   
1962/2297 KB   
1966/2297 KB   
1970/2297 KB   
1974/2297 KB   
1978/2297 KB   
1982/2297 KB   
1986/2297 KB   
1990/2297 KB   
1994/2297 KB   
1998/2297 KB   
2002/2297 KB   
2006/2297 KB   
2010/2297 KB   
2014/2297 KB   
2018/2297 KB   
2022/2297 KB   
2026/2297 KB   
2030/2297 KB   
2034/2297 KB   
2038/2297 KB   
2042/2297 KB   
2046/2297 KB   
2050/2297 KB   
2054/2297 KB   
2058/2297 KB   
2062/2297 KB   
2066/2297 KB   
2070/2297 KB   
2073/2297 KB   
2077/2297 KB   
2081/2297 KB   
2085/2297 KB   
2089/2297 KB   
2093/2297 KB   
2097/2297 KB   
2101/2297 KB   
2105/2297 KB   
2109/2297 KB   
2113/2297 KB   
2117/2297 KB   
2121/2297 KB   
2125/2297 KB   
2129/2297 KB   
2133/2297 KB   
2137/2297 KB   
2141/2297 KB   
2145/2297 KB   
2149/2297 KB   
2153/2297 KB   
2157/2297 KB   
2161/2297 KB   
2165/2297 KB   
2169/2297 KB   
2173/2297 KB   
2177/2297 KB   
2181/2297 KB   
2185/2297 KB   
2189/2297 KB   
2193/2297 KB   
2197/2297 KB   
2201/2297 KB   
2205/2297 KB   
2209/2297 KB   
2213/2297 KB   
2217/2297 KB   
2221/2297 KB   
2225/2297 KB   
2229/2297 KB   
2233/2297 KB   
2237/2297 KB   
2241/2297 KB   
2245/2297 KB   
2249/2297 KB   
2253/2297 KB   
2257/2297 KB   
2261/2297 KB   
2265/2297 KB   
2269/2297 KB   
2273/2297 KB   
2277/2297 KB   
2281/2297 KB   
2285/2297 KB   
2289/2297 KB   
2293/2297 KB   
2297/2297 KB   
   
Downloaded: 
http://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.3.16/byte-buddy-1.3.16.jar
 (2297 KB at 10071.2 KB/sec)
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ joshua ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.2.1:process (default) @ joshua ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ joshua ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ joshua ---
[INFO] Compiling 264 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
joshua ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 319 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
joshua ---
[INFO] Compiling 41 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ joshua ---
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng/2.19.1/surefire-testng-2.19.1.pom
3/3 KB 
 
Downloaded: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng/2.19.1/surefire-testng-2.19.1.pom
 (3 KB at 21.8 KB/sec)
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng-utils/2.19.1/surefire-testng-utils-2.19.1.pom
3/3 KB   
 
Downloaded: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng-utils/2.19.1/surefire-testng-utils-2.19.1.pom
 (3 KB at 40.9 KB/sec)
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng/2.19.1/surefire-testng-2.19.1.jar
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng-utils/2.19.1/surefire-testng-utils-2.19.1.jar
4/42 KB   
8/42 KB   
12/42 KB   
16/42 KB   
20/42 KB   
24/42 KB   
28/42 KB   
32/42 KB   
36/42 KB   
40/42 KB   
42/42 KB   
3/27 KB   42/42 KB   
7/27 KB   42/42 KB   
11/27 KB   42/42 KB   
15/27 KB   42/42 KB   
19/27 KB   42/42 KB   
23/27 KB   42/42 KB   
27/27 KB   42/42 KB   
27/27 KB   42/42 KB   
  
Downloaded: 
http://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-testng/2.19.1/surefire-testng-2.19.1.jar
 (42 KB at 531.5

[jira] [Closed] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2016-06-19 Thread Thamme Gowda (JIRA) 

 [ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thamme Gowda closed JOSHUA-277.
---
Resolution: Won't Fix

> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
>   at 
>

[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path

2016-06-19 Thread Thamme Gowda (JIRA) 

[ 
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338604#comment-15338604
 ] 

Thamme Gowda commented on JOSHUA-277:
-

Thanks for the link, [~post]. I will follow this tutorial.


> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
>  Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html 
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried 
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate 
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero 
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en   
>   --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class 
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  [CHANGED]
>   dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>   
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
>  [NOT FOUND]
>   
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>  
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
>  --tunedir 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner 
> mert --decoder 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
>  --decoder-config 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
>  --decoder-output-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
>  --decoder-log-file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
>  --iterations 10 --metric 'BLEU 4 closest'
>   JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file 
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
>   at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
>   at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
>   at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
>   ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
>   at 
>