[GitHub] incubator-joshua pull request #81: Fix: memory overflow in tuning with large...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/81 ---
[GitHub] incubator-joshua pull request #81: Fix: memory overflow in tuning with large...
GitHub user thammegowda opened a pull request: https://github.com/apache/incubator-joshua/pull/81 Fix: memory overflow in tuning with larger language models tuner needs more memory when larger language models are used. Even though we allocate more memory in the training pipeline script the value is ignored at one place in pipeline. This PR fixes it by exporting env var, and passing it to the JVM at the right place You can merge this pull request into a Git repository by running: $ git pull https://github.com/thammegowda/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/81.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #81 commit cad86aa7512f2829c8a964f2ccc3f69c7cd2f1c7 Author: Thamme GowdaDate: 2018-03-06T01:13:27Z Fix: memory overflow in tuning when larger language models are used, the tuner needs more memory. Even though we allocate more memory in the training pipeline script the value is ignored at one place in pipeline. This patch fixes it by exporting env var, and passing it to the JVM ---
[GitHub] incubator-joshua pull request #80: Code Reformatting and output of string le...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/80 ---
[GitHub] incubator-joshua pull request #80: Code Reformatting and output of string le...
GitHub user thammegowda opened a pull request: https://github.com/apache/incubator-joshua/pull/80 Code Reformatting and output of string length ratios one new number in the eval output - sentence length (number of words) ratios between output and reference. However, to interpret the existing code, I had to indent the code correctly. There are a lot of edits, but most of them are due to code reformatting - no functional change. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thammegowda/incubator-joshua code-clean Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/80.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #80 commit 36e3a5d39702e2f139afa7f0bd67220beef4cfa5 Author: Thamme GowdaDate: 2017-10-25T18:46:28Z code clean and reformat commit 7eae36dbae9fe64ef929800cb7160e3739f16f77 Author: Thamme Gowda Date: 2017-10-25T19:03:06Z Output Ratio of Lengths of Strings ---
[GitHub] incubator-joshua pull request #79: JOSHUA-324 renamed LICENSE, removed minif...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/79 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #79: JOSHUA-324 renamed LICENSE, removed minif...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/79 JOSHUA-324 renamed LICENSE, removed minification of license in bootst⦠â¦rap.min.css Small patch to make detection of the license easier in this minified file. I believe this was flagged during our release review. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/79.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #79 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #78: JOSHUA-324 Included correct license for O...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/78 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #73: JOSHUA-316 run_bundler.py returning JOB F...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/73 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #77: Tweaked Dockerfile to contain only jre 1....
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/77 Tweaked Dockerfile to contain only jre 1.8 and to use mvn package Just some small changes here. I found when creating files with the previous version that is was often using java 1.7 for some reason. Changed the base image partially to get an image that already has maven, and partly to make sure on jre 1.8 is installed. Also switched to mvn package. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/77.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #77 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #76: Release
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/76 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #75: JOSHUA-322 extract-1best script reference...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/75 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #74: JOSHUA-321 Add JOSHUA env to ./bin/bleu a...
GitHub user lewismc opened a pull request: https://github.com/apache/incubator-joshua/pull/74 JOSHUA-321 Add JOSHUA env to ./bin/bleu and ./bin/extract-1best bash scripts This issue addresses https://issues.apache.org/jira/browse/JOSHUA-321 You can merge this pull request into a Git repository by running: $ git pull https://github.com/lewismc/incubator-joshua JOSHUA-321 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/74.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #74 commit 2f3e752969a2bacc17aa029595c9a56b93331e9e Author: Lewis John McGibbneyDate: 2016-11-09T23:31:30Z JOSHUA-321 Add JOSHUA env to ./bin/bleu and ./bin/extract-1best bash scripts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #73: JOSHUA-316 run_bundler.py returning JOB F...
GitHub user lewismc opened a pull request: https://github.com/apache/incubator-joshua/pull/73 JOSHUA-316 run_bundler.py returning JOB FAILED (return code 1) TypeError: memoryview: a bytes-like object is required, not 'str' This issue addresses https://issues.apache.org/jira/browse/JOSHUA-316 This issue was a complete PITA. The non ascii character in run_bundler.py README template was a frigging big PITA. Done now folks :) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lewismc/incubator-joshua JOSHUA-316 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/73.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #73 commit d137f2970d1eace1dcec576d287712cd5b88 Author: Lewis John McGibbneyDate: 2016-10-26T02:31:10Z JOSHUA-316 run_bundler.py returning JOB FAILED (return code 1) TypeError: memoryview: a bytes-like object is required, not 'str' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #71: JOSHUA-290 - separate OSGi artifact for J...
GitHub user tteofili opened a pull request: https://github.com/apache/incubator-joshua/pull/71 JOSHUA-290 - separate OSGi artifact for Joshua, based on 7 branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-joshua JOSHUA-290-branch7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit 7fa72ed49d0d0acb98d27115161464d450a6930d Author: Tommaso TeofiliDate: 2016-10-10T10:28:07Z JOSHUA-290 - separate OSGi artifact for Joshua, based on 7 branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
Github user tteofili closed the pull request at: https://github.com/apache/incubator-joshua/pull/69 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
Github user tteofili commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/69#discussion_r82572784 --- Diff: src/main/java/org/apache/joshua/decoder/package-info.java --- @@ -23,4 +23,7 @@ * of any actual decoding algorithm. Rather, such code is in * child packages of this package. */ -package org.apache.joshua.decoder; \ No newline at end of file +@Version("0.1.0") +package org.apache.joshua.decoder; + +import org.osgi.annotation.versioning.Version; --- End diff -- sounds good to me, I'll close this one and create a new PR for merging into 7 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
Github user KellenSunderland commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/69#discussion_r8215 --- Diff: src/main/java/org/apache/joshua/decoder/package-info.java --- @@ -23,4 +23,7 @@ * of any actual decoding algorithm. Rather, such code is in * child packages of this package. */ -package org.apache.joshua.decoder; \ No newline at end of file +@Version("0.1.0") +package org.apache.joshua.decoder; + +import org.osgi.annotation.versioning.Version; --- End diff -- I would vote for putting it in the 7 branch actually. @mjpost what do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
Github user hsaputra commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/69#discussion_r81881247 --- Diff: src/main/java/org/apache/joshua/decoder/package-info.java --- @@ -23,4 +23,7 @@ * of any actual decoding algorithm. Rather, such code is in * child packages of this package. */ -package org.apache.joshua.decoder; \ No newline at end of file +@Version("0.1.0") +package org.apache.joshua.decoder; + +import org.osgi.annotation.versioning.Version; --- End diff -- This should go to `maven-multi-module` branch I suppose? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #70: validate the original implementation of S...
GitHub user cocoxu opened a pull request: https://github.com/apache/incubator-joshua/pull/70 validate the original implementation of SARI You can merge this pull request into a Git repository by running: $ git pull https://github.com/cocoxu/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/70.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #70 commit 307d5f2c57e0367a6deb500aec79490b36b9090e Author: Wei Coco XuDate: 2016-09-30T15:11:53Z validate the original implementation of SARI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
Github user tteofili commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/69#discussion_r81326566 --- Diff: src/main/java/org/apache/joshua/decoder/package-info.java --- @@ -23,4 +23,7 @@ * of any actual decoding algorithm. Rather, such code is in * child packages of this package. */ -package org.apache.joshua.decoder; \ No newline at end of file +@Version("0.1.0") +package org.apache.joshua.decoder; + +import org.osgi.annotation.versioning.Version; --- End diff -- Sure, creating a separate module for OSGi sounds good to me. Note however that the annotation dependency is at compile time, that shouldn't be needed at runtime, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #69: JOSHUA-290 - provide Joshua as a bundle
GitHub user tteofili opened a pull request: https://github.com/apache/incubator-joshua/pull/69 JOSHUA-290 - provide Joshua as a bundle You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-joshua JOSHUA-290 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/69.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #69 commit f0596b4efe63ccbb1ff9748b7d6d826d52218484 Author: Tommaso TeofiliDate: 2016-09-29T12:39:32Z JOSHUA-290 - provide Joshua as a bundle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #67: JOSHUA-299 Move regression tests to prope...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/67 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #68: KenLM JNI optimization
Github user kpu commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/68#discussion_r80882244 --- Diff: jni/kenlm_wrap.cc --- @@ -76,15 +76,16 @@ class EqualIndex : public std::binary_function{ typedef std::unordered_set Lookup; /** - * A Chart bundles together a unordered_multimap that maps ChartState signatures to a single - * object instantiated using a pool. This allows duplicate states to avoid allocating separate - * state objects at multiple places throughout a sentence, and also allows state to be shared - * across KenLMs for the same sentence. Multimap is used to avoid hash collisions which can - * return incorrect results, and cause out-of-bounds lookups when multiple KenLMs are in use. + * A Chart bundles together a vector holding CharStates and an unordered_set of StateIndexes + * which provides a mapping between StateIndexes and the positions of ChartStates in the vector. + * This allows for duplicate states to avoid allocating separate state objects at multiple places + * throughout a sentence. */ class Chart { public: -Chart() : lookup_(1000, HashIndex(vec_), EqualIndex(vec_)) {} +Chart(long* ngramBuffer) : --- End diff -- Good practice to use explicit with single-argument constructors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #68: KenLM JNI optimization
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/68 KenLM JNI optimization I've cleaned up the code from the previous PR (#65) that had the @kpu's perf increase. I've removed the code that was specific to the estimate call as we didn't see a performance increase with it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua DirectBuffersRemoveEst Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/68.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #68 commit 9ea7eebf0164d1676f633b441bd952eaa20b0760 Author: Kellen SunderlandDate: 2016-09-15T17:06:04Z Convert to a DirectBuffer to transfer ngrams during probRule commit c8d8a65b9352e51e777965994dae7f9337b08def Author: Kellen Sunderland Date: 2016-09-15T17:31:21Z Converted estimateRule to also make use of DirectBuffer. Reduced number of array copies in probRule. Removed sentence from estimate method signature (as it was unused). Created an abstraction in the KenLMPool class to hide details of underlying ByteBuffer Indexing. Fixed Test givenKenLm_whenQueryingWithState_thenStateAndProbReturned commit d9c3d7ecf069a6a0339b911b9defb8ce31ebb1f1 Author: Kellen Sunderland Date: 2016-09-27T15:31:37Z Remove uneeded modifications for estimate in KenLM commit e9f4f5b1468364a658f90c168e2b8ec69c3fa48e Author: Kellen Sunderland Date: 2016-09-27T16:29:50Z Explicitly bind KenLMs to LmPool objects --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #66: Merge pull request #1 from apache/7_confs...
Github user mjpost closed the pull request at: https://github.com/apache/incubator-joshua/pull/66 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #65: This PR is a first attempt to minimize ng...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/65 This PR is a first attempt to minimize ngram array copies by utilizing DirectBuffers. This is a work in progress but posted so that Matt + Ken can review (and do some speed comparisons). You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/65.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #65 commit eadfdf3d09bea43fa3af6c664a7f8944b0cc8f86 Author: Kellen SunderlandDate: 2016-09-15T17:06:04Z Convert to a DirectBuffer to transfer ngrams during probRule commit 3917d8503659dd4ac3a55b3181ecb07f32b067da Author: Kellen Sunderland Date: 2016-09-15T17:30:22Z Merge branch 'master' of https://github.com/apache/incubator-joshua --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #64: Fixed Rescoring test
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/64 Fixed Rescoring test You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/64.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #64 commit 695feb33293423ab6fca1d8f03b2eb0f4fe9393a Author: Hieber, FelixDate: 2016-09-15T15:16:56Z Fixed Rescoring test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #63: Fixed more tests in 7
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/63 Fixed more tests in 7 You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7_new Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/63.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #63 commit 1091b9e8d49165a2ec9f20b62f3a15b37ffb4dda Author: Hieber, FelixDate: 2016-09-15T14:57:01Z Fixed SourceAnnotations Test for 7 commit 0d99e4d47f868d83afcb34379fbf984a5452dca1 Author: Hieber, Felix Date: 2016-09-15T14:58:52Z Fixed TooLongTest for 7 commit c6cd09748312f0338c1002f2adb19fd8308fcb7d Author: Hieber, Felix Date: 2016-09-15T15:05:10Z Merge remote-tracking branch 'upstream/7' into 7_new --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #62: Fixed NumTranslationOptions test and adde...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/62 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #62: Fixed NumTranslationOptions test and adde...
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/62 Fixed NumTranslationOptions test and added deterministic output order⦠⦠of feature vectors You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/62.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #62 commit bc050e3b5c99f71c4d5945dc9852a13e825958f7 Author: Hieber, FelixDate: 2016-09-15T14:48:58Z Fixed NumTranslationOptions test and added deterministic output order of feature vectors --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #61: Fixed NAry, Constrained deocoding, and Le...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/61 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #61: Fixed NAry test for 7
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/61 Fixed NAry test for 7 You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/61.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #61 commit f08573b5f64a37afcc824df233db8a7e8719f7e9 Author: Hieber, FelixDate: 2016-09-15T12:28:13Z Fixed NAry test for 7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #60: Moved several regression tests to unit te...
GitHub user michael-aloys opened a pull request: https://github.com/apache/incubator-joshua/pull/60 Moved several regression tests to unit tests (3). You can merge this pull request into a Git repository by running: $ git pull https://github.com/michael-aloys/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/60.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #60 commit 9cbc045fe8b058b3fac028180ad99390b9fbf709 Author: Michael A. HedderichDate: 2016-09-14T17:45:50Z Moved regression test decoder/num_translation_options to unit test. Regenerated and split gold output into several files to be used by different test methods. commit 931a67d78c8f7de69695b400ed8c2db046d86784 Author: Michael A. Hedderich Date: 2016-09-15T09:46:32Z Moved regression test decoder/oov-list to unit test. Regenerated gold output and cleaned up the directory. Copied the lm from n-ary to remove dependency to other test resource directory. commit 11b87c38ccfcdbe967d46b1f06834a5c5355fed3 Author: Michael A. Hedderich Date: 2016-09-15T09:58:33Z Moved regression test decoder/rescoring to unit test. Regenerated gold output and cleaned up the directory. Copied the lm from constrained to remove dependency to other test resource directory. commit d9f34aa24b2eb884947a7c0113819cfdb0534559 Author: Michael A. Hedderich Date: 2016-09-15T10:00:04Z Removed resources of unused regression test decoder/segment-oovs commit 32f27536978879ecfb7be69c820cdfa33e7fc7f5 Author: Michael A. Hedderich Date: 2016-09-15T10:34:44Z Moved regression test decoder/source-annotations to unit test. Regenerated gold output and cleaned up the directory. commit 17ecec13e942e08855074611df5c9d36cb5b57d7 Author: Michael A. Hedderich Date: 2016-09-15T10:53:58Z Moved regression test decoder/too-long to unit test. Removed resources as they are no longer needed. commit 99b475b3ffe08f2bfcc8105e906293dbd9d1c12f Author: Michael A. Hedderich Date: 2016-09-15T11:00:38Z Moved regression test decoder/tree-output to unit test. Regenerated gold output and cleaned up the directory. commit d9bb3e5a841998c0bb8b87018c06ef28b46176c4 Author: Michael A. Hedderich Date: 2016-09-15T11:03:04Z Removed resources of unused regression test decoder/phrase/include-align-index. commit b429cc7d8802a67c0eb948339984d5c7f5bef24e Author: Michael A. Hedderich Date: 2016-09-15T12:09:51Z Moved regression test decoder/phrase/unique-hypothesis to unit test. Replaced broken soft-links with actual files and cleaned up the directory. commit b9e6ffac00fa15906f6642107b7b8140e6465cfe Author: Michael A. Hedderich Date: 2016-09-15T12:19:30Z Moved regression test decoder/target-bigram to unit test. Cleaned up the corresponding directory. commit b09081aede83f3fbe895bf0f4e4b79577a4c70a6 Author: Michael A. Hedderich Date: 2016-09-15T12:23:42Z Added missing license texts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #59: Fixed NoGrammarTest for 7
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/59 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #59: Fixed NoGrammarTest for 7
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/59 Fixed NoGrammarTest for 7 You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/59.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #59 commit c2747787ae222a36362be2de8cc2e5e26fcc01fb Author: Hieber, FelixDate: 2016-09-15T11:43:02Z Fixed NoGrammarTest for 7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #55: Refactoring of ComputeNodeResult into a s...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/55 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #58: Moved new unit tests (former regression t...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/58 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #58: Moved new unit tests (former regression t...
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/58 Moved new unit tests (former regression tests) to correct location. The previous merge from master placed the new regression tests from Michael into the wrong location. This moves them to joshua-core/src/ so that they actually get run. Now they are failing. Can someone please investigate? You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/58.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #58 commit ee7398f7d0c921cf1aaf7937a370a41d31431033 Author: Hieber, FelixDate: 2016-09-14T21:19:00Z moved new unit tests (former regression tests) to correct location in joshua 7 (joshua-core/src) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #57: Moved several regression tests to unit te...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/57 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #56: Moved several regression tests to unit te...
GitHub user michael-aloys opened a pull request: https://github.com/apache/incubator-joshua/pull/56 Moved several regression tests to unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/michael-aloys/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/56.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #56 commit 74d800657377a05ded552cb5b2b26c46a5b6bc52 Author: Michael A. HedderichDate: 2016-09-14T12:32:35Z Moved regression test decoder/do-not-crash to unit test and refactored code to reuse utility test functions. commit 0871742c7b6512260053969af75c1c2a91ae323a Author: Michael A. Hedderich Date: 2016-09-14T12:58:56Z Moved regression test decoder/empty-test to unit test. commit 99c791d1d54702061aafeb7678e88ae40521b419 Author: Michael A. Hedderich Date: 2016-09-14T13:12:46Z Moved regression test decoder/constrained to unit test. Cleaned up the corresponding directory. commit dc55f07a9440fe32afc5658fed3ac6acecb22cd1 Author: Michael A. Hedderich Date: 2016-09-14T13:15:50Z Added missing license in Unit test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #55: Refactoring of ComputeNodeResult into a s...
GitHub user tdomhan opened a pull request: https://github.com/apache/incubator-joshua/pull/55 Refactoring of ComputeNodeResult into a static method. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tdomhan/incubator-joshua computenoderesult Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/55.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #55 commit dd82f081b9519764237b6fbbab3e84a397b9594c Author: Tobias DomhanDate: 2016-09-14T13:12:54Z Refactoring of ComputeNodeResult into a static method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #54: Moved regression tests bn-en/packed and b...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/54 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #54: Moved regression tests bn-en/packed and b...
GitHub user michael-aloys opened a pull request: https://github.com/apache/incubator-joshua/pull/54 Moved regression tests bn-en/packed and bn-en/samt to unit test Moved the shell script regression tests from bn-en/packed and bn-en/samt to a unit test class. Cleaned up the corresponding resource directory. Regenerated gold output with "%c %s" format. Should also fix the failing Travis CI on Mac without KenLM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/michael-aloys/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/54.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #54 commit e199d8514b52bf6842f19afb2afb51d218c18c3a Author: Michael A. HedderichDate: 2016-09-14T09:50:33Z Moved regression tests bn-en/packed and bn-en/samt to unit test Moved the shell script regression tests from bn-en/packed and bn-en/samt to a unit test class. Also cleaned up the corresponding resource directory. Regenerated gold output with "%c %s" format. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #51: Changes to improve performance of KenLM
Github user KellenSunderland closed the pull request at: https://github.com/apache/incubator-joshua/pull/51 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #50: fix basedir path for java.library.path
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/50 fix basedir path for java.library.path makes unit tests run with kenlm again You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/50.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #50 commit f0883156c10a521cc22ee0698b7c15456256b4bb Author: Felix HieberDate: 2016-09-12T14:08:54Z fix basedir path for java.library.path --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #24: Maven multi-module project layout proposa...
Github user logogin closed the pull request at: https://github.com/apache/incubator-joshua/pull/24 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #49: 7 with master
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/49 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #42: Fix various issues related to resources, ...
Github user maxthomas commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/42#discussion_r76825646 --- Diff: src/main/java/org/apache/joshua/decoder/ff/lm/buildin_lm/TrieLM.java --- @@ -264,66 +215,65 @@ public static void main(String[] args) throws IOException { int n = Integer.valueOf(args[2]); LOG.info("N-gram order will be {}", n); -Scanner scanner = new Scanner(new File(args[1])); +try (Scanner scanner = new Scanner(new File(args[1]));) { --- End diff -- I personally prefer the semicolon: * it's necessary if other statements/resources are added to the autoclose block * it's easier to determine the actual method call when there are many parens towards the end but I don't mind either way. What would be nice is a commit that applies the format across the entire project. A ton of lines have trailing whitespace or tabs or whatever. I'n going to file an issue on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #42: Fix various issues related to resources, ...
Github user KellenSunderland commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/42#discussion_r76822903 --- Diff: src/main/java/org/apache/joshua/decoder/ff/lm/buildin_lm/TrieLM.java --- @@ -264,66 +215,65 @@ public static void main(String[] args) throws IOException { int n = Integer.valueOf(args[2]); LOG.info("N-gram order will be {}", n); -Scanner scanner = new Scanner(new File(args[1])); +try (Scanner scanner = new Scanner(new File(args[1]));) { --- End diff -- Nitpick: would it be ok to remove the semicolon from the autocloseable lines? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #42: Fix various issues related to resources, ...
Github user KellenSunderland commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/42#discussion_r76822076 --- Diff: src/main/java/org/apache/joshua/decoder/ff/lm/buildin_lm/TrieLM.java --- @@ -188,62 +188,13 @@ public TrieLM(ArpaFile arpaFile) throws FileNotFoundException { @Override - protected double logProbabilityOfBackoffState_helper( - int[] ngram, int order, int qtyAdditionalBackoffWeight - ) { + protected double logProbabilityOfBackoffState_helper(int[] ngram, int order, int qtyAdditionalBackoffWeight) { throw new UnsupportedOperationException("probabilityOfBackoffState_helper undefined for TrieLM"); } @Override protected float ngramLogProbability_helper(int[] ngram, int order) { - -//float logProb = (float) -JoshuaConfiguration.lm_ceiling_cost;//Float.NEGATIVE_INFINITY; // log(0.0f) -float backoff = 0.0f; // log(1.0f) - -int i = ngram.length - 1; -int word = ngram[i]; -i -= 1; - -int nodeID = ROOT_NODE_ID; - -while (true) { - - { -long key = Bits.encodeAsLong(nodeID, word); -if (logProbs.containsKey(key)) { -// logProb = logProbs.get(key); - backoff = 0.0f; // log(0.0f) -} - } - - if (i < 0) { -break; - } - - { -long key = Bits.encodeAsLong(nodeID, ngram[i]); - -if (children.containsKey(key)) { - nodeID = children.get(key); - - backoff += backoffs.get(nodeID); - - i -= 1; - -} else { - break; -} - } - -} - -//double result = logProb + backoff; -//if (result < -JoshuaConfiguration.lm_ceiling_cost) { -// result = -JoshuaConfiguration.lm_ceiling_cost; -//} -// -//return result; -return (Float) null; +throw new UnsupportedOperationException(); --- End diff -- Good change, but I'd just like to note this is an example of https://en.wikipedia.org/wiki/Interface_segregation_principle that we should think about cleaning up one day. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #47: Fix Travis-CI by adding a step to install...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/47 Fix Travis-CI by adding a step to install ant on OSX It looks like the download-deps.sh now requires ant to build BerkleyLM? If so feel free to merge this, it should make sure the OSX image has ant installed before building. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/47.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #47 commit 0b447595ae51ab3d2956d60748962a092e192286 Author: Kellen SunderlandDate: 2016-08-30T08:38:35Z Fix Travis-CI by adding a step to install ant on OSX --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #46: This update merges from master with threa...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/46 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #46: The update merges from master and applies...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/46 The update merges from master and applies some refactoring. This PR has the merge from master with the Threading change. On top of that merge I've got one commit that reduces the number of constructors for JoshuaDecoder and gets rid of a bunch of now unused arguments. Because I've removed these arguments, this is a breaking change. I'm assuming this is ok for the 7 branch? I didn't include the breaking change in the pull on the master branch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua 7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/46.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #46 commit 5ed36b09484ec18fe00a8cb20af9d0e1ba1ca4e7 Author: Lewis John McGibbneyDate: 2016-08-24T02:36:09Z Update examples README formatting and links. commit 0744ebf56906dbe70292737cd50a39652407869d Author: Lewis John McGibbney Date: 2016-08-24T03:01:06Z Update examples README pipeline invocation parameters commit 762d588efc820cc7e6b98fce454b9254d8d15518 Author: Lewis John McGibbney Date: 2016-08-28T01:17:36Z Add Brew install badge to README commit d1c9c074544d72ee5335bdd83fe415b45098ab08 Author: Kellen Sunderland Date: 2016-08-29T08:08:10Z JOSHUA-285 JOSHUA-296 Refactored threading in order to properly propagate failures and remove custom code commit 0bb293295e3670c7449815941566578facd247e9 Author: Kellen Sunderland Date: 2016-08-29T11:53:53Z Renamed DecoderThread to DecoderTask commit 6d8f684836ddc25e40bb32f91d24d3f6e5eb745b Author: Kellen Sunderland Date: 2016-08-29T11:58:52Z Renamed Translations class to TranslationResponseStream commit 2a458c04f644861fccc3db521579e72ef335c656 Author: Kellen Sunderland Date: 2016-08-29T13:46:35Z Merge remote-tracking branch 'origin' into 7 commit 0e87046c9aa933b365638a88dce694b938ab84ef Author: Kellen Sunderland Date: 2016-08-29T13:55:50Z Merge branch 'master' of https://github.com/KellenSunderland/incubator-joshua into 7 commit f90cf3e4d947b32a27ef8f3d5f51d06677fd8d65 Author: Kellen Sunderland Date: 2016-08-25T09:50:43Z Addressed todos from previous commits. Refactored entry points in Joshua to remove redundancy, this is a breaking change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #45: Refactored the threading code in the Josh...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/45 Refactored the threading code in the JoshuaDecoder class. These commits do some refactoring and reworking of the threading code in JoshuaDecoder. The principal motivation for this refactor is to reduce our use on custom code, and utilize libraries. It should also improve readability and reduce the potential for bugs as we've removed a large chunk of fairly complex code. Note: I've seen some errors with tests due to static Vocab issues with my new test. Will look into fixing this (and will fix right away if required for this PR). You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/45.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #45 commit d1c9c074544d72ee5335bdd83fe415b45098ab08 Author: Kellen SunderlandDate: 2016-08-29T08:08:10Z JOSHUA-285 JOSHUA-296 Refactored threading in order to properly propagate failures and remove custom code commit 0bb293295e3670c7449815941566578facd247e9 Author: Kellen Sunderland Date: 2016-08-29T11:53:53Z Renamed DecoderThread to DecoderTask commit 6d8f684836ddc25e40bb32f91d24d3f6e5eb745b Author: Kellen Sunderland Date: 2016-08-29T11:58:52Z Renamed Translations class to TranslationResponseStream --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/d4fdbfd88bab99e244d3ed1fc9cff4ba5e6d124c#commitcomment-18760447 I think maybe no, since all the feature functions are loaded using reflection. This could be a method available only on StatefulFF. Or maybe there is a better way to do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/d4fdbfd88bab99e244d3ed1fc9cff4ba5e6d124c#commitcomment-18742703 The whole static global feature index is a mess. You've gotten rid of the dense feature indices on the 7 branch. What if we get rid of the global state index as well? We could fix this in initializeFeatureFunctions, with the following test: ... this.featureFunctions.add(feature); int stateIndex = 0; if (feature instanceof StatefulFF) { feature.setStateIndex(stateIndex); stateIndex++; } Thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #39: JOSHUA-301 - Add findbugs plugin.
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/39 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #44: Moved test case files
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/44 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #44: Moved test case files
GitHub user mjpost opened a pull request: https://github.com/apache/incubator-joshua/pull/44 Moved test case files Moved test cases files from resources/ to src/test/resources. Also fixed bug in LmOovFeatureTest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-joshua JOSHUA-299 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/44.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #44 commit 20e6bf4b069d261718f819a72c9e9b2c62bcbb26 Author: Matt PostDate: 2016-08-22T22:03:53Z Moved test file locations from resources/ to src/test/resources commit 4812fedee89d3a6dac4e8333a1d62242f3b5ac9c Author: Matt Post Date: 2016-08-23T02:36:09Z missed one commit d4fdbfd88bab99e244d3ed1fc9cff4ba5e6d124c Author: Matt Post Date: 2016-08-23T02:57:06Z bugfix: resetting global decoder state We are getting random failures in LmOovFeatureTest.java, but only when run as a group. I printed the stack trace around the error and noticed it's a call to getting the state index in ComputeNodeResult, with a state index of 1, which shouldn't happen because only one LM is loaded. So it seems that the bug is cause by some earlier test not calling Decoder.resetGlobalState() in cleanup. So I put a call to that in the constructor. This passes the tests but I'm not sure it's correct, and it's definitely not the right way to go about things. The ideal way to solve this is to get rid of global state. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #43: Phrase-based decoder rewrite
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/43 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #43: Phrase-based decoder rewrite
GitHub user mjpost opened a pull request: https://github.com/apache/incubator-joshua/pull/43 Phrase-based decoder rewrite The phrase-based decoder used to add nonterminals to every phrase-based rule, treating all such rules as left-branching ones. This was a hassle because everything had to be converted, e.g., after extracting from Thrax. Now, the phrase tables have no nonterminals on the source and target sides. Instead, glue rules are used. This means this is not backwards compatible. Phrase-based language packs will have to be recompiled, but this needs to be done anyway. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-joshua JOSHUA-284 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/43.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #43 commit dcc7e7ee72228de08b70003a49344c2614eaedbe Author: Matt PostDate: 2016-08-16T22:13:06Z large commit converting phrase-based decoding to new rule format Not working yet, but much of the code is redone and future estimates are being computed correctly commit 32504c47bbc90b3fd4a8d02298b9758fa8126a44 Author: Matt Post Date: 2016-08-16T22:13:50Z updated scripts to work with the new format commit 48a9aad7873b969230aad90d6e0c61e13ae2d2b4 Author: Matt Post Date: 2016-08-16T22:14:15Z repacked the grammar commit dac822d15145614c33f5fb12d2797e1f91825bb3 Author: Matt Post Date: 2016-08-17T10:23:57Z missed file in commit commit b1ec62711d15f3b692b6a7026752123f75522f6e Author: Matt Post Date: 2016-08-17T10:24:07Z enabled test commit 1022699cc744fa9fbc21f4b19122f51e3985a371 Author: Matt Post Date: 2016-08-17T10:24:46Z temporary commenting-out of very verbose output commit 2e746c1864ca7eb6be27f2fca3ab258c9ebe7adb Author: Matt Post Date: 2016-08-19T18:14:18Z changed order of assert() args commit 048b2e30f849de3f1ac82e6017ea2aab299f6b8d Author: Matt Post Date: 2016-08-19T18:15:18Z removed RHS nonterminal commit af4ef88d5a6a6a1cc4167ec421b4b6bd1a91dc0a Author: Matt Post Date: 2016-08-19T18:15:36Z added derived directories commit 9b73d6147a61580058cc57c86c1f623f44b7452a Author: Matt Post Date: 2016-08-19T18:16:47Z build two nodes over terminal productions commit 5719c8cff728499bffd1053462351340f1d91353 Author: Matt Post Date: 2016-08-19T18:17:21Z fixed distortion computation to work with new format code now produces a translation on my test case, though it's not the correct one commit eb00223870c7683cf8e557ab689a1979fb36ec1d Author: Matt Post Date: 2016-08-20T00:43:58Z converted from span -> separate i, j commit 473b3016562677671f70a19cd15d67a2bc1a5c83 Author: Matt Post Date: 2016-08-20T00:44:14Z off-by-one error in computing future estimates commit 574cb36b5e1b610e37eda81d6d76b4318c141a4c Author: Matt Post Date: 2016-08-20T00:44:44Z bugfix: this is (probably) supposed to return the pruning estimate commit 16d5647bee30345ffa56b5b7d5bebc1021afa3fa Author: Matt Post Date: 2016-08-20T00:45:12Z fixed computation of distortion commit 36cde50ba37df9c9b2ead6b063ac5935e3dd253d Author: Matt Post Date: 2016-08-20T13:30:42Z moved comparator into Candidate commit 49dbf8cbaf2f1e0c648f8eb705ab3887aa06b039 Author: Matt Post Date: 2016-08-20T13:31:18Z removed nonterminals from OOV rules commit e3b60ca9a7fea7d25a8533b630a1a66d29349a6f Author: Matt Post Date: 2016-08-21T11:53:26Z minor cleanup of assignment logic commit 293db94c2853f7dc15bd6fecdf3b39bd3a4b4965 Author: Matt Post Date: 2016-08-21T11:53:41Z Bug fix in reporting inside cost â everything now works commit 0e49bc537b05549930802bf6c187b849c4c67adb Author: Matt Post Date: 2016-08-21T11:55:55Z added debug-joshua which sets debugging and uses classes instead of the jar This means you can run the command-line version while in Eclipse without having to rebuild the jar file (which is time-consuming). commit d6820c6f3bc41ca87dfff4a8ed18172de4f849e6 Author: Matt Post Date: 2016-08-21T12:01:17Z removed debugging output commit cd3ff0c6d0d2ad959cd4f292d9ee02f4e7da8b0a Author: Matt Post Date: 2016-08-21T12:01:38Z removed alignments from test, created new test with alignments (currently not working...) commit 25d28fe2ce32a4b130a4412e982d6e16d5af8afc Author: Matt Post
[GitHub] incubator-joshua pull request #42: Fix various issues related to resources, ...
GitHub user maxthomas opened a pull request: https://github.com/apache/incubator-joshua/pull/42 Fix various issues related to resources, warnings, and copy-pasted code. I didn't make an issue for this, because it hits on a lot of topics that would be tough to summarize in one issue. But, let me know if you'd like me to do so. The gist of this commit is code cleanup. Resource leaks are closed, warnings are resolved, and the beginning of refactoring the 10K lines of copy/pasted code throughout MERT, MIRA, etc. was started. I plan on working on this further on the 7 branch. But, because many resources weren't being closed explicitly (I found the finalize override in LineReader __extremely__ questionable), I figured I'd submit this on master right away. Feel free to ignore/postpone/whatever You can merge this pull request into a Git repository by running: $ git pull https://github.com/maxthomas/incubator-joshua fix-warnings-resources Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/42.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #42 commit 980f735853a9403823501a66ff857e0c5763ae52 Author: max thomasDate: 2016-08-18T19:04:52Z Fix a number of issues: - Reader now implements autocloseable - Close various leaks from LineReader - LineReader no longer implements custom finalize(). Resources should be explicitly closed when no longer needed. The compiler helps with this. - Start refactoring copy/pasted code into a new type: ExistingUTF8EncodedTextFile. There is so much use of text files that this should really have its own type. - Fix warnings about unused fields, unused methods - Delete some old/legacy/unused classes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #41: Major refactoring of features/rules/gramm...
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/41 Major refactoring of features/rules/grammars in Joshua Major refactoring of core decoder components (Rule.java, FeatureVector.java and grammars). The core idea of this change is to simplify feature handling inside Joshua. Please note that this change is NOT backwards compatible. The following changes were made: - No distinction between sparse and dense features inside the decoder anymore. Each feature stored at the rule is 'owned' by the grammar that contains the rule. An 'owned' feature simply means that its name is prepended with the owner string: 0=0.2 becomes _0=0.2. This applies to both dense features (features that occur at every rule), as well as sparse features. Please note that the old prefix 'tm_' is no longer used. - Having only one type of feature, a revised version of FeatureVector.java was built that is greatly simplified. It is basically a HashMap of FeatureId (typed as ints) to feature values. FeatureIds are created/hashed by the new global mapping FeatureMap.java, which maintains a bidirectional mapping between feature ids and feature names. This also allowed getting rid of storing feature names in the vocabulary. - The simplified FeatureVector cause removal of all 'reportDenseFeatures'/'getNumDenseFeatures' method in the decoder and the grammar interface. - The tradition but very obscure way of flipping the sign of dense features but not sparse features was removed. The feature value in the decoder is now just the value as you see it stored at the rule. - The Rule class was changed to adhere to object-oriented principles. It now has only one constructor that requires all of its dependencies and these can not be changed later. This forces Rule creators to finalize the dependencies (deciding on an owner of the rule and the hashing of the feature vector). - Also the unused concept of the precomputableCost in a rule was removed. Rules still 'cache' their estimated cost. - The various Grammar and MemoryBasedBatchGrammar constructors were unified and a lot of old obscure code was removed. - Due to the change above, the PhraseModel feature function that fires feature values for features stored at rules is greatly simplified. - As featureVectors at Rules are final and have to have an owner, feature sharing across multiple grammars would need to be handled by a separate feature function implementation which is transparent. This commit also updates all existing (and enabled) Unit tests which also pass. Existing regression tests do NOT work in this commit since many of the grammars are packed and would need to be re-packed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua 7_features Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/41.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #41 commit 20afddf85263f3def242f721728ac148ef143ad5 Author: Felix HieberDate: 2016-08-17T11:52:39Z Major refactoring of core decoder components (Rule.java, FeatureVector.java and grammars). The core idea of this change is to simplify feature handling inside Joshua. Please note that this change is NOT backwards compatible. The following changes were made: - No distinction between sparse and dense features inside the decoder anymore. Each feature stored at the rule is 'owned' by the grammar that contains the rule. An 'owned' feature simply means that its name is prepended with the owner string: 0=0.2 becomes _0=0.2. This applies to both dense features (features that occur at every rule), as well as sparse features. Please note that the old prefix 'tm_' is no longer used. - Having only one type of feature, a revised version of FeatureVector.java was built that is greatly simplified. It is basically a HashMap of FeatureId (typed as ints) to feature values. FeatureIds are created/hashed by the new global mapping FeatureMap.java, which maintains a bidirectional mapping between feature ids and feature names. This also allowed getting rid of storing feature names in the vocabulary. - The simplified FeatureVector cause removal of all 'reportDenseFeatures'/'getNumDenseFeatures' method in the decoder and the grammar interface. - The tradition but very obscure way of flipping the sign of dense features but not sparse features was removed. The feature value in the decoder is now just the value as you see it stored at the rule. - The Rule class was changed to adhere to object-oriented principles. It now has only one constructor that requires all of its dependencies and these can not be changed later. This forces Rule creators to
[GitHub] incubator-joshua pull request #40: JOSHUA-302 - Remove dependency on concurr...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/40 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #40: JOSHUA-302 - Remove dependency on concurr...
GitHub user maxthomas opened a pull request: https://github.com/apache/incubator-joshua/pull/40 JOSHUA-302 - Remove dependency on concurrent/concurrent package You can merge this pull request into a Git repository by running: $ git pull https://github.com/maxthomas/incubator-joshua JOSHUA-302 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/40.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #40 commit 19fe7561aa516b1a7eba63473e8d26111a708c33 Author: Max ThomasDate: 2016-08-17T15:32:20Z JOSHUA-302 - Remove dependency on concurrent/concurrent package; replace instances with JDK library builtins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #39: JOSHUA-301 - Add findbugs plugin.
GitHub user maxthomas opened a pull request: https://github.com/apache/incubator-joshua/pull/39 JOSHUA-301 - Add findbugs plugin. Currently this can be run with 'mvn findbugs:check'. There are a ton of warnings, however, which currently fail the build if this is run as part of the maven build. Perhaps once fixed this auto-build failure can be added in, but at present there are too many warnings to fix at present. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maxthomas/incubator-joshua JOSHUA-301 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/39.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #39 commit dccb48bcc0ab76faaa3b9f3e58823b5fa66ea983 Author: Max ThomasDate: 2016-08-17T15:13:37Z JOSHUA-301 - Add findbugs plugin. Currently this can be run with 'mvn findbugs:check'. There are a ton of warnings, however, which currently fail the build. Perhaps once fixed this auto-build failure can be added in, but at present there are too many warnings to fix immediately. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/3e2e053db9cafcdc5b4885a0f6fa5e54a63bc468#commitcomment-18672288 No --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user fhieber commented on the pull request: https://github.com/apache/incubator-joshua/commit/3e2e053db9cafcdc5b4885a0f6fa5e54a63bc468#commitcomment-18672173 In src/test/java/org/apache/joshua/system/LmOovFeatureTest.java: In src/test/java/org/apache/joshua/system/LmOovFeatureTest.java on line 69: is this main required? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #38: JOSHUA-294 - Remove CommandLineParser cla...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/38 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #37: JOSHUA-292 - Add travis CI badge to READM...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/37 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #37: JOSHUA-292 - Add travis CI badge to READM...
GitHub user maxthomas opened a pull request: https://github.com/apache/incubator-joshua/pull/37 JOSHUA-292 - Add travis CI badge to README.md You can merge this pull request into a Git repository by running: $ git pull https://github.com/maxthomas/incubator-joshua JOSHUA-292 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit d0e43a6586cb501fa1d7053e4f313409aa373608 Author: Max ThomasDate: 2016-08-12T00:19:16Z JOSHUA-292 - Add travis CI badge to README.md --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #36: Demo
Github user hsaputra commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/36#discussion_r73449163 --- Diff: src/main/java/org/apache/joshua/server/ServerThread.java --- @@ -225,15 +225,18 @@ private void handleMetadata(String meta, JSONMessage message) { String source = argTokens[1]; String target = argTokens[2]; String featureStr = ""; + String alignmentStr = ""; if (argTokens.length > 3) featureStr = argTokens[3]; - + if (argTokens.length > 4) +alignmentStr = " ||| " + argTokens[4]; --- End diff -- What is this `alignmentStr` argument for ? Wonder if we could add test for this new argument --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/0663a9c7f704e1fc911d1792910ca9337840553b#commitcomment-18511694 @KellenSunderland do you have a minute to look at this? I thought this had passed CI tests, but Travis [is now telling me](https://travis-ci.org/apache/incubator-joshua/builds/148722346) that it didn't. It runs fine on my OS X machine and on a Linux machine I have access to, but seems to fail the generic test here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #36: Demo
GitHub user mjpost opened a pull request: https://github.com/apache/incubator-joshua/pull/36 Demo Made some improvements to the Joshua demo under demo/. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjpost/incubator-joshua demo Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/36.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #36 commit c465e71530eecead797f73c06da25f1c0cff28ed Author: Matt PostDate: 2016-08-02T13:08:42Z fiddling with demo commit 1038a144468b48408f3eca1122efb3c8f84e4537 Author: Matt Post Date: 2016-08-02T13:41:39Z quick add rules for OOVs, format fiddling commit aedeafd0df1a30ecf686193cd9446ba7e569e84f Author: Matt Post Date: 2016-08-02T16:14:22Z HTTP server now projects case commit 3387b16befb1e8121a05e7b3a13d041bdeb7260b Author: Matt Post Date: 2016-08-02T16:19:44Z fixed to work with detokenized output commit ca6fc49dc853ea07189e65b2df9e77f36bbfd7dd Author: Matt Post Date: 2016-08-02T16:20:03Z denormalizes output commit 84301b9f04b3ffeecf35432309b1ca762c692313 Author: Matt Post Date: 2016-08-02T16:20:10Z remove sentence marker commit 81b33d2218e51bc70f3ef42c2f0cd3c0d3059340 Author: Matt Post Date: 2016-08-02T16:38:27Z added alignment point in custom rules --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #35: Add OSX building to .travis.yml
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/35 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #33: Refactored unit tests to all use TestNG, ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/33 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #35: Add OSX building to .travis.yml
GitHub user KellenSunderland reopened a pull request: https://github.com/apache/incubator-joshua/pull/35 Add OSX building to .travis.yml You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua travis2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/35.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #35 commit 91323783fd1a8a35d1b0a7908ebf9c6401cc4f4a Author: Kellen SunderlandDate: 2016-07-29T09:55:00Z Add OSX building to .travis.yml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #35: Add OSX building to .travis.yml
Github user KellenSunderland closed the pull request at: https://github.com/apache/incubator-joshua/pull/35 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #35: Add OSX building to .travis.yml
GitHub user KellenSunderland reopened a pull request: https://github.com/apache/incubator-joshua/pull/35 Add OSX building to .travis.yml You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua travis2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/35.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #35 commit 7d6da91852c630fe338cc2b8bca279ae449762a8 Author: Kellen SunderlandDate: 2016-07-29T09:55:00Z Add OSX building to .travis.yml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #34: Enable basic travis build
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/34 Enable basic travis build This should get us started with Travis. I'll open a PR to see if we're at least passing with a basic config, then add some more build steps later. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua travis Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/34.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #34 commit b9b43531e84a7922b3a3126754d32a46fae3b9eb Author: Kellen SunderlandDate: 2016-07-28T15:45:30Z Enable basic travis build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #33: Refactored unit tests to all use TestNG, ...
GitHub user KellenSunderland opened a pull request: https://github.com/apache/incubator-joshua/pull/33 Refactored unit tests to all use TestNG, removed no longer applicable⦠⦠tests, refactored KenLM in order to make handling upstream errors easier You can merge this pull request into a Git repository by running: $ git pull https://github.com/KellenSunderland/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/33.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #33 commit 701d746d9670964b77b7f8fd26b030deccf25f0c Author: Kellen SunderlandDate: 2016-07-12T17:34:52Z Refactored unit tests to all use TestNG, removed no longer applicable tests, refactored KenLM in order to make handling upstream errors easier --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #32: JOSHUA-286 - Replace old joshua-decoder.o...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/32#discussion_r72566062 --- Diff: scripts/support/make-release.sh --- @@ -40,7 +40,7 @@ echo "Bundling up joshua-$version" [[ ! -d release ]] && mkdir release rm -f joshua-$version && ln -s $JOSHUA joshua-$version -wget -r http://joshua-decoder.org/ +wget -r http://joshua.apache.org/6.0/ --- End diff -- Again, this site does not exist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #32: JOSHUA-286 - Replace old joshua-decoder.o...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/32#discussion_r72566047 --- Diff: CHANGELOG --- @@ -139,7 +139,7 @@ They include: - Significantly improved and expanded documentation (both user and developer) - See http://joshua-decoder.org/4.0 or ./joshua-decoder.org/4.0/index.html (local mirror) + See http://apache.joshua.org/4.0 or ./apache.joshua.org/4.0/index.html (local mirror) --- End diff -- There is no such site as http://apache.joshua.org, it should be http://joshua.apache.org, also the CHANGELOG should not be altered at all as this indicates changes to previous releases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #32: JOSHUA-286 - Replace old joshua-decoder.o...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/incubator-joshua/pull/32#discussion_r72566005 --- Diff: .gitignore --- @@ -42,6 +42,7 @@ doxygen_*.tmp .cachepipe joshua-decoder.org +joshua.apache.org --- End diff -- both of these can be removed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #32: JOSHUA-286 - Replace old joshua-decoder.o...
GitHub user john-hewitt opened a pull request: https://github.com/apache/incubator-joshua/pull/32 JOSHUA-286 - Replace old joshua-decoder.org links with joshua.apache.org - Update links to documentation and support to reflect the move to Apache. - keep Gitignore entry for old website was kept to keep the repo clean. - Update links to the git repo as well. - old pages in the `docs` folder unchanged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/john-hewitt/incubator-joshua JOSHUA-286 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/32.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #32 commit 36a58e75e5deb71bdeed2980e740502fe3d516c2 Author: John HewittDate: 2016-07-28T02:57:04Z Replace old joshua-decoder.org links with joshua.apache.org Updating links to documentation and support to reflect the move to Apache. Gitignore entry was kept to keep the repo clean. References to the old git repo were updated as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #31: Refactored unit tests to all use TestNG, ...
Github user ThePasswordIsPassword closed the pull request at: https://github.com/apache/incubator-joshua/pull/31 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #29: Added option to fire a dense language mod...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/29 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #29: Added option to fire a dense language mod...
GitHub user fhieber opened a pull request: https://github.com/apache/incubator-joshua/pull/29 Added option to fire a dense language model oov indicator feature from LanguageModelFF. By default it is turned off. To activate, specify '-oov_feature' in the lm line in joshua.config. This OOV feature is different from the OOV rules in Joshua grammars/phrase tables. It indicates an OOV in the language model which typically has a much larger vocabulary. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhieber/incubator-joshua lm_feature Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit b3063a4674222a3419eac17e310b1d2cb693fbc8 Author: Felix HieberDate: 2016-06-24T07:13:52Z Added option to fire a dense language model oov indicator feature from LanguageModelFF. By default it is turned off. To activate, specify '-oov_feature' in the lm line in joshua.config --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user fhieber commented on the pull request: https://github.com/apache/incubator-joshua/commit/748cf32f956d3fd5772f62e023d0cc8d70a7dcad#commitcomment-17995315 No its totally fine. Thanks for the fix! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user fhieber commented on the pull request: https://github.com/apache/incubator-joshua/commit/3fb809c444c7855255b51080c368071bc508be1f#commitcomment-17995297 I am fine with testng. Its equally easy to run in eclipse with the plugin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/3fb809c444c7855255b51080c368071bc508be1f#commitcomment-17995203 I don't have any experience that would drive this decision, but I agree we should decide on one. Currently, eclipse tells me there are 42 junit tests and 94 (87 passing / 2 failing / 5 skipped) testng tests. It also gives me a tool to convert junit tests to testng. @fhieber, does that change your mind? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/748cf32f956d3fd5772f62e023d0cc8d70a7dcad#commitcomment-17994955 No problem, it didn't get triggered until I had a small case where I actually asked for the alignments on output. I suppose we could also have fixed this by passing UKNOWN_OWNER_ID, we can switch it if you prefer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user fhieber commented on the pull request: https://github.com/apache/incubator-joshua/commit/748cf32f956d3fd5772f62e023d0cc8d70a7dcad#commitcomment-17994947 Ah, right! Thanks! Should have looked more closely. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/commit/748cf32f956d3fd5772f62e023d0cc8d70a7dcad#commitcomment-17994847 Sure â UNKNOWN_OWNER is a String, so the constructor that gets called is [this one](https://github.com/apache/incubator-joshua/blob/master/src/main/java/org/apache/joshua/decoder/ff/tm/Rule.java#L157), which interprets the last argument as an alignment, not an owner. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #:
Github user fhieber commented on the pull request: https://github.com/apache/incubator-joshua/commit/748cf32f956d3fd5772f62e023d0cc8d70a7dcad#commitcomment-17994753 Hi Matt, Can you comment how this was a bug and what this change changes? Using the Rule constructor without an owner will just assign the Unknown owner id to it in the constructor itself afaik. So what should the owner be here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #28: Migrated the Dockerfile to use new Maven ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/28 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #26: Owner Ids are maintained in separate mapp...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/26 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #25: ClassLMs: fixed a bug with class-based lm...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-joshua/pull/25 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #28: Migrated the Dockerfile to use new Maven ...
GitHub user dameikle opened a pull request: https://github.com/apache/incubator-joshua/pull/28 Migrated the Dockerfile to use new Maven build You can merge this pull request into a Git repository by running: $ git pull https://github.com/LogicalSpark/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/28.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #28 commit f78127902d11c3aa0590159aa48d63970f02b95c Author: David MeikleDate: 2016-06-22T17:54:59Z Migrated the Dockerfile to use new Maven build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-joshua pull request #27: Adding the CHRF metric for tuning
GitHub user antot opened a pull request: https://github.com/apache/incubator-joshua/pull/27 Adding the CHRF metric for tuning This pull request adds the CHRF metric for tuning. You can merge this pull request into a Git repository by running: $ git pull https://github.com/antot/incubator-joshua master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-joshua/pull/27.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #27 commit 52c83d5930c74e15186c5710b9b47b97d2a9230b Author: antotDate: 2016-06-21T16:12:40Z Adding the CHRF metric for tuning commit 32a5003e671bfd781a5bf774cc48d72443fc2c1e Author: antot Date: 2016-06-21T16:24:34Z Adding the CHRF metric for tuning --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---