Jenkins build is back to normal : joshua_master #95

[Apache Jenkins Server]

See 

[jira] [Commented] (JOSHUA-299) Move regression tests to proper unit tests

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/JOSHUA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432062#comment-15432062
 ] 

Lewis John McGibbney commented on JOSHUA-299:
-

I'll scope this issue tomorrow [~post] and see if I can get a PR together.

> Move regression tests to proper unit tests
> --
>
> Key: JOSHUA-299
> URL: https://issues.apache.org/jira/browse/JOSHUA-299
> Project: Joshua
>  Issue Type: Bug
>Reporter: Matt Post
>Assignee: Lewis John McGibbney
> Fix For: 6.1
>
>
> Many of the regression tests (test*.sh under src/test/resources) have been 
> moved to proper unit tests, but this move should be completed, and the 
> regression tests should be deleted. This should be done for 6.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (JOSHUA-299) Move regression tests to proper unit tests

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JOSHUA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney reassigned JOSHUA-299:
---

Assignee: Lewis John McGibbney

> Move regression tests to proper unit tests
> --
>
> Key: JOSHUA-299
> URL: https://issues.apache.org/jira/browse/JOSHUA-299
> Project: Joshua
>  Issue Type: Bug
>Reporter: Matt Post
>Assignee: Lewis John McGibbney
> Fix For: 6.1
>
>
> Many of the regression tests (test*.sh under src/test/resources) have been 
> moved to proper unit tests, but this move should be completed, and the 
> regression tests should be deleted. This should be done for 6.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JOSHUA-301) Add findbugs plugin to maven build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/JOSHUA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432060#comment-15432060
 ] 

Hudson commented on JOSHUA-301:
---

FAILURE: Integrated in Jenkins build joshua_master #94 (See 
[https://builds.apache.org/job/joshua_master/94/])
JOSHUA-301 - Add findbugs plugin. (max: rev 
d235f9b09ec41698c6d9784dcc05926ee55a03a1)
* (edit) pom.xml


> Add findbugs plugin to maven build
> --
>
> Key: JOSHUA-301
> URL: https://issues.apache.org/jira/browse/JOSHUA-301
> Project: Joshua
>  Issue Type: Sub-task
>  Components: build
>Reporter: Max Thomas
>Priority: Minor
> Fix For: 6.1
>
>
> Add the findbugs maven plugin to the build. This will allow the build to 
> report on issues found via the popular Findbugs static analysis tool. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: joshua_master #94

[Apache Jenkins Server]

See 

Changes:

[max] JOSHUA-301 - Add findbugs plugin.

[post] Moved test file locations from resources/ to src/test/resources

[post] missed one

[post] bugfix: resetting global decoder state

--
[...truncated 740 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

[jira] [Commented] (JOSHUA-301) Add findbugs plugin to maven build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/JOSHUA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432052#comment-15432052
 ] 

ASF GitHub Bot commented on JOSHUA-301:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-joshua/pull/39


> Add findbugs plugin to maven build
> --
>
> Key: JOSHUA-301
> URL: https://issues.apache.org/jira/browse/JOSHUA-301
> Project: Joshua
>  Issue Type: Sub-task
>  Components: build
>Reporter: Max Thomas
>Priority: Minor
> Fix For: 6.1
>
>
> Add the findbugs maven plugin to the build. This will allow the build to 
> report on issues found via the popular Findbugs static analysis tool. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-joshua pull request #39: JOSHUA-301 - Add findbugs plugin.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-joshua/pull/39


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-joshua pull request #44: Moved test case files

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-joshua/pull/44


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-joshua pull request #44: Moved test case files

[mjpost]

GitHub user mjpost opened a pull request:

https://github.com/apache/incubator-joshua/pull/44

Moved test case files

Moved test cases files from resources/ to src/test/resources. Also fixed 
bug in LmOovFeatureTest.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-joshua JOSHUA-299

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-joshua/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44


commit 20e6bf4b069d261718f819a72c9e9b2c62bcbb26
Author: Matt Post 
Date:   2016-08-22T22:03:53Z

Moved test file locations from resources/ to src/test/resources

commit 4812fedee89d3a6dac4e8333a1d62242f3b5ac9c
Author: Matt Post 
Date:   2016-08-23T02:36:09Z

missed one

commit d4fdbfd88bab99e244d3ed1fc9cff4ba5e6d124c
Author: Matt Post 
Date:   2016-08-23T02:57:06Z

bugfix: resetting global decoder state

We are getting random failures in LmOovFeatureTest.java, but only when run 
as a group. I printed the stack trace around the error and noticed it's a call 
to getting the state index in ComputeNodeResult, with a state index of 1, which 
shouldn't happen because only one LM is loaded. So it seems that the bug is 
cause by some earlier test not calling Decoder.resetGlobalState() in cleanup. 
So I put a call to that in the constructor. This passes the tests but I'm not 
sure it's correct, and it's definitely not the right way to go about things. 
The ideal way to solve this is to get rid of global state.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-joshua issue #43: Phrase-based decoder rewrite

[mjpost]

Github user mjpost commented on the issue:

https://github.com/apache/incubator-joshua/pull/43
  
Followup: this appears to have something to do with a failed global state 
cleanup from a prior test run. Fixed (temporary at least) at commit 
d4fdbfd88bab99e244d3ed1fc9cff4ba5e6d124c.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (JOSHUA-284) Phrase-based decoding changes

[Matt Post (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JOSHUA-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Post resolved JOSHUA-284.
--
Resolution: Fixed

> Phrase-based decoding changes
> -
>
> Key: JOSHUA-284
> URL: https://issues.apache.org/jira/browse/JOSHUA-284
> Project: Joshua
>  Issue Type: Bug
>Reporter: Matt Post
>Assignee: Matt Post
> Fix For: 6.1
>
>
> Joshua's phrase-based decoding creates a lot of complications in the pipeline.
> Currently, phrase-based rules are simply left-branching Hiero rules. This 
> means that, prior to packing or loading, rules have to have a nonterminal 
> prepended to them. For example, Thrax will extract
> [X] ||| yo quiero ||| i want ||| ...
> This has to be changed to
> [X] ||| [X,1] yo quiero ||| [X,1] yo quiero ||| ...
> This means, for one, that phrase tables share a format but are specific to 
> either the hiero or phrase-based decoder.
> Another problem is that the alignments have to be adjusted when packing 
> grammars from Moses or Thrax format, since a symbol is being added. 
> Basically, this choice introduces a host of incompatibilities that require 
> special handling.
> A better idea would be to change the phrase-based decoder a bit so that, 
> instead of using left-branching phrase rules, it made use of proper glue 
> rules, the same way Hiero does. The advantages are:
> - both formalisms would use the same format
> - both formalisms would have a glue grammar
> - there should be no impact in running time



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-joshua pull request #43: Phrase-based decoder rewrite

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-joshua/pull/43


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-joshua issue #43: Phrase-based decoder rewrite

[mjpost]

Github user mjpost commented on the issue:

https://github.com/apache/incubator-joshua/pull/43
  
More information:

- On OS X, remove $JOSHUA/lib/libken.dylib, run "mvn test" → FAILURE
- Same, but run the individual test from Eclipse → PASSES
- Same, but run all tests from Eclipse → PASSES


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-joshua issue #43: Phrase-based decoder rewrite

[mjpost]

Github user mjpost commented on the issue:

https://github.com/apache/incubator-joshua/pull/43
  
Strange: the LmOovFeatureTest is failing again when KenLM is not available, 
but now it's on a Mac OS X instance, and not on the Linux one. I'm tempted to 
disable this test because its failure is so random; on the other hand, maybe 
the random failure is indicative of a deeper problem. Can anyone repeat this 
failure? I can repeat it on my laptop when I remove $JOSHUA/lib/libken.dylib.

@KellenSunderland @fhieber 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (JOSHUA-282) %S output format doesn't remove

[Matt Post (JIRA)]

[ 
https://issues.apache.org/jira/browse/JOSHUA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431656#comment-15431656
 ] 

Matt Post commented on JOSHUA-282:
--

This is fixed with commit bf12adc8b8e130c9f9addc69f47e9cf7e0774f72, which will 
be merged into master for 6.1.

> %S output format doesn't remove 
> ---
>
> Key: JOSHUA-282
> URL: https://issues.apache.org/jira/browse/JOSHUA-282
> Project: Joshua
>  Issue Type: Bug
>Reporter: Matt Post
>Assignee: Matt Post
> Fix For: 6.1
>
>
> Using -output-format %S with the phrase-based decoder prevents removal of the 
>  and  tags.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (JOSHUA-282) %S output format doesn't remove

[Matt Post (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JOSHUA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Post resolved JOSHUA-282.
--
Resolution: Fixed

> %S output format doesn't remove 
> ---
>
> Key: JOSHUA-282
> URL: https://issues.apache.org/jira/browse/JOSHUA-282
> Project: Joshua
>  Issue Type: Bug
>Reporter: Matt Post
>Assignee: Matt Post
> Fix For: 6.1
>
>
> Using -output-format %S with the phrase-based decoder prevents removal of the 
>  and  tags.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-joshua pull request #43: Phrase-based decoder rewrite

[mjpost]

GitHub user mjpost opened a pull request:

https://github.com/apache/incubator-joshua/pull/43

Phrase-based decoder rewrite

The phrase-based decoder used to add nonterminals to every phrase-based 
rule, treating all such rules as left-branching ones. This was a hassle because 
everything had to be converted, e.g., after extracting from Thrax. Now, the 
phrase tables have no nonterminals on the source and target sides. Instead, 
glue rules are used.

This means this is not backwards compatible. Phrase-based language packs 
will have to be recompiled, but this needs to be done anyway.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-joshua JOSHUA-284

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-joshua/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #43


commit dcc7e7ee72228de08b70003a49344c2614eaedbe
Author: Matt Post 
Date:   2016-08-16T22:13:06Z

large commit converting phrase-based decoding to new rule format

Not working yet, but much of the code is redone and future estimates are 
being computed correctly

commit 32504c47bbc90b3fd4a8d02298b9758fa8126a44
Author: Matt Post 
Date:   2016-08-16T22:13:50Z

updated scripts to work with the new format

commit 48a9aad7873b969230aad90d6e0c61e13ae2d2b4
Author: Matt Post 
Date:   2016-08-16T22:14:15Z

repacked the grammar

commit dac822d15145614c33f5fb12d2797e1f91825bb3
Author: Matt Post 
Date:   2016-08-17T10:23:57Z

missed file in commit

commit b1ec62711d15f3b692b6a7026752123f75522f6e
Author: Matt Post 
Date:   2016-08-17T10:24:07Z

enabled test

commit 1022699cc744fa9fbc21f4b19122f51e3985a371
Author: Matt Post 
Date:   2016-08-17T10:24:46Z

temporary commenting-out of very verbose output

commit 2e746c1864ca7eb6be27f2fca3ab258c9ebe7adb
Author: Matt Post 
Date:   2016-08-19T18:14:18Z

changed order of assert() args

commit 048b2e30f849de3f1ac82e6017ea2aab299f6b8d
Author: Matt Post 
Date:   2016-08-19T18:15:18Z

removed RHS nonterminal

commit af4ef88d5a6a6a1cc4167ec421b4b6bd1a91dc0a
Author: Matt Post 
Date:   2016-08-19T18:15:36Z

added derived directories

commit 9b73d6147a61580058cc57c86c1f623f44b7452a
Author: Matt Post 
Date:   2016-08-19T18:16:47Z

build two nodes over terminal productions

commit 5719c8cff728499bffd1053462351340f1d91353
Author: Matt Post 
Date:   2016-08-19T18:17:21Z

fixed distortion computation to work with new format

code now produces a translation on my test case, though it's not the 
correct one

commit eb00223870c7683cf8e557ab689a1979fb36ec1d
Author: Matt Post 
Date:   2016-08-20T00:43:58Z

converted from span -> separate i, j

commit 473b3016562677671f70a19cd15d67a2bc1a5c83
Author: Matt Post 
Date:   2016-08-20T00:44:14Z

off-by-one error in computing future estimates

commit 574cb36b5e1b610e37eda81d6d76b4318c141a4c
Author: Matt Post 
Date:   2016-08-20T00:44:44Z

bugfix: this is (probably) supposed to return the pruning estimate

commit 16d5647bee30345ffa56b5b7d5bebc1021afa3fa
Author: Matt Post 
Date:   2016-08-20T00:45:12Z

fixed computation of distortion

commit 36cde50ba37df9c9b2ead6b063ac5935e3dd253d
Author: Matt Post 
Date:   2016-08-20T13:30:42Z

moved comparator into Candidate

commit 49dbf8cbaf2f1e0c648f8eb705ab3887aa06b039
Author: Matt Post 
Date:   2016-08-20T13:31:18Z

removed nonterminals from OOV rules

commit e3b60ca9a7fea7d25a8533b630a1a66d29349a6f
Author: Matt Post 
Date:   2016-08-21T11:53:26Z

minor cleanup of assignment logic

commit 293db94c2853f7dc15bd6fecdf3b39bd3a4b4965
Author: Matt Post 
Date:   2016-08-21T11:53:41Z

Bug fix in reporting inside cost — everything now works

commit 0e49bc537b05549930802bf6c187b849c4c67adb
Author: Matt Post 
Date:   2016-08-21T11:55:55Z

added debug-joshua which sets debugging and uses classes instead of the jar

This means you can run the command-line version while in Eclipse without 
having to rebuild the jar file (which is time-consuming).

commit d6820c6f3bc41ca87dfff4a8ed18172de4f849e6
Author: Matt Post 
Date:   2016-08-21T12:01:17Z

removed debugging output

commit cd3ff0c6d0d2ad959cd4f292d9ee02f4e7da8b0a
Author: Matt Post 
Date:   2016-08-21T12:01:38Z

removed alignments from test, created new test with alignments (currently 
not working...)

commit 25d28fe2ce32a4b130a4412e982d6e16d5af8afc
Author: Matt Post