[jira] [Commented] (JOSHUA-312) Alignment is never cached

2016-09-22 Thread Lewis John McGibbney (JIRA) 

[ 
https://issues.apache.org/jira/browse/JOSHUA-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514037#comment-15514037
 ] 

Lewis John McGibbney commented on JOSHUA-312:
-

Thanks [~post], to be specific, if I run a pipeline and it crashes... say at 
Thrax execution (another ticket coming for that very soon), then I attempt to 
re-run the exact same pipeline, then the alignment is always re-done even 
though material is cached in the working directory. 
Does that make sense? If not then I can paste evidence of it happening as code 
samples below. Please let me know. Thanks

> Alignment is never cached
> -
>
> Key: JOSHUA-312
> URL: https://issues.apache.org/jira/browse/JOSHUA-312
> Project: Joshua
>  Issue Type: Improvement
>  Components: alignment
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.2
>
>
> Say if a pipeline fails after alignment. The alignment result is never cached 
> and it becomes necessary to undertake alignment... again!
> We should investigate the process for caching alignments as it would really 
> speed up rerunning end-to-end pipelines for large input datasets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (JOSHUA-312) Even though alignment is cached, it is always re-done in pipeline re-execution

2016-09-22 Thread Lewis John McGibbney (JIRA) 

 [ 
https://issues.apache.org/jira/browse/JOSHUA-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated JOSHUA-312:

Summary: Even though alignment is cached, it is always re-done in pipeline 
re-execution  (was: Alignment is never cached)

> Even though alignment is cached, it is always re-done in pipeline re-execution
> --
>
> Key: JOSHUA-312
> URL: https://issues.apache.org/jira/browse/JOSHUA-312
> Project: Joshua
>  Issue Type: Improvement
>  Components: alignment
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.2
>
>
> Say if a pipeline fails after alignment. The alignment result is never cached 
> and it becomes necessary to undertake alignment... again!
> We should investigate the process for caching alignments as it would really 
> speed up rerunning end-to-end pipelines for large input datasets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: moses2 vs. joshua

2016-09-22 Thread Tommaso Teofili 
+1, let's go for the release as soon as possible,

Tommaso

Il giorno gio 22 set 2016 alle ore 18:29 Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> ha scritto:

> Sounds totally cool! Great work Matt
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect, Instrument Software and Science Data Systems Section (398)
> Manager, Open Source Projects Formulation and Development Office (8212)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Director, Information Retrieval and Data Science Group (IRDS)
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> WWW: http://irds.usc.edu/
> ++
>
>
>
> On 9/22/16, 6:09 AM, "Matt Post"  wrote:
>
> Hi folks,
>
> I have finished the comparison. Here you can find graphs for ar-en and
> ru-en. The ground-up rewrite of Moses is
> about 2x–3x faster than Joshua.
>
> http://imgur.com/a/FcIbW
>
> One implication (untested) is that we are likely as fast as or faster
> than Moses.
>
> We could brainstorm things to do to close this gap. I'd be much
> happier with 2x or even 1.5x than with 3x, and I bet we could narrow this
> down. But I'd like to get the 6.1 release out of the way, first, so I'm
> pushing this off to next month. Sound cool?
>
> matt
>
>
> > On Sep 19, 2016, at 6:26 AM, Matt Post  wrote:
> >
> > I can't believe I did this, but I mis-colored one of the hiero
> lines, and the Numbers legend doesn't show the line type. If you reload the
> dropbox file, it's fixed now. The difference is about 3x for both. Here's
> the table.
> >
> > Threads
> > Joshua
> > Moses2
> > Joshua (hiero)
> > Moses2 (hiero)
> > Phrase rate
> > Hiero rate
> > 1
> > 178
> > 65
> > 2116
> > 1137
> > 2.74
> > 1.86
> > 2
> > 109
> > 42
> > 1014
> > 389
> > 2.60
> > 2.61
> > 4
> > 78
> > 29
> > 596
> > 213
> > 2.69
> > 2.80
> > 6
> > 72
> > 25
> > 473
> > 154
> > 2.88
> > 3.07
> >
> > I'll put the models together and share them later today. This was on
> a 6-core machine and I agree it'd be nice to test with something much
> higher.
> >
> > matt
> >
> >
> >> On Sep 19, 2016, at 5:33 AM, kellen sunderland <
> kellen.sunderl...@gmail.com > wrote:
> >>
> >> Do we just want to store these models somewhere temporarily?  I've
> got a OneDrive account and could share the models from there (as long as
> they're below 500GBs or so).
> >>
> >> On Mon, Sep 19, 2016 at 11:32 AM, kellen sunderland <
> kellen.sunderl...@gmail.com > wrote:
> >> Very nice results.  I think getting to within 25% of a optimized
> c++ decoder from a Java decoder is impressive.  Great that Hieu has put in
> the work to make moses2 so fast as well, that gives organizations two quite
> nice decoding engines to choose from, both with reasonable performance.
> >>
> >> Matt: I had a question about the x axis here.  Is that number of
> threads?  We should be scaling more or less linearly with the number of
> threads, is that the case here?  If you post the models somewhere I can
> also do a quick benchmark on a machine with a few more cores.
> >>
> >> -Kellen
> >>
> >>
> >> On Mon, Sep 19, 2016 at 10:53 AM, Tommaso Teofili <
> tommaso.teof...@gmail.com > wrote:
> >> Il giorno sab 17 set 2016 alle ore 15:23 Matt Post  > ha
> >> scritto:
> >>
> >>> I'll ask Hieu; I don't anticipate any problems. One potential
> problem is
> >>> that that models occupy about 15--20 GB; do you think Jenkins
> would host
> >>> this?
> >>>
> >>
> >> I'm not sure, can such models be downloaded and pruned at runtime,
> or do
> >> they need to exist on the Jenkins machine ?
> >>
> >>
> >>>
> >>> (ru-en grammars still packing, results will probably not be in
> until much
> >>> later today)
> >>>
> >>> matt
> >>>
> >>>
>  On Sep 17, 2016, at 3:19 PM, Tommaso Teofili <
> tommaso.teof...@gmail.com >
> >>> wrote:
> 
>  Hi Matt,
> 
>  I think it'd be really valuable if we could be able to repeat the
> same
>  tests (given parallel corpus is available) in the future, any
> chance you
>  can share script / code to do that ? We may even consider adding a
> >>> Jenkins
>  job dedicated to continuously monitor performances as we w

Re: moses2 vs. joshua

2016-09-22 Thread Mattmann, Chris A (3980) 
Sounds totally cool! Great work Matt

++
Chris Mattmann, Ph.D.
Chief Architect, Instrument Software and Science Data Systems Section (398)
Manager, Open Source Projects Formulation and Development Office (8212)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Director, Information Retrieval and Data Science Group (IRDS)
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++
 


On 9/22/16, 6:09 AM, "Matt Post"  wrote:

Hi folks,

I have finished the comparison. Here you can find graphs for ar-en and 
ru-en. The ground-up rewrite of Moses is 
about 2x–3x faster than Joshua.

http://imgur.com/a/FcIbW

One implication (untested) is that we are likely as fast as or faster than 
Moses.

We could brainstorm things to do to close this gap. I'd be much happier 
with 2x or even 1.5x than with 3x, and I bet we could narrow this down. But I'd 
like to get the 6.1 release out of the way, first, so I'm pushing this off to 
next month. Sound cool?

matt


> On Sep 19, 2016, at 6:26 AM, Matt Post  wrote:
> 
> I can't believe I did this, but I mis-colored one of the hiero lines, and 
the Numbers legend doesn't show the line type. If you reload the dropbox file, 
it's fixed now. The difference is about 3x for both. Here's the table.
> 
> Threads
> Joshua
> Moses2
> Joshua (hiero)
> Moses2 (hiero)
> Phrase rate
> Hiero rate
> 1
> 178
> 65
> 2116
> 1137
> 2.74
> 1.86
> 2
> 109
> 42
> 1014
> 389
> 2.60
> 2.61
> 4
> 78
> 29
> 596
> 213
> 2.69
> 2.80
> 6
> 72
> 25
> 473
> 154
> 2.88
> 3.07
> 
> I'll put the models together and share them later today. This was on a 
6-core machine and I agree it'd be nice to test with something much higher.
> 
> matt
> 
> 
>> On Sep 19, 2016, at 5:33 AM, kellen sunderland 
mailto:kellen.sunderl...@gmail.com>> wrote:
>> 
>> Do we just want to store these models somewhere temporarily?  I've got a 
OneDrive account and could share the models from there (as long as they're 
below 500GBs or so).
>> 
>> On Mon, Sep 19, 2016 at 11:32 AM, kellen sunderland 
mailto:kellen.sunderl...@gmail.com>> wrote:
>> Very nice results.  I think getting to within 25% of a optimized c++ 
decoder from a Java decoder is impressive.  Great that Hieu has put in the work 
to make moses2 so fast as well, that gives organizations two quite nice 
decoding engines to choose from, both with reasonable performance.
>> 
>> Matt: I had a question about the x axis here.  Is that number of 
threads?  We should be scaling more or less linearly with the number of 
threads, is that the case here?  If you post the models somewhere I can also do 
a quick benchmark on a machine with a few more cores. 
>> 
>> -Kellen
>> 
>> 
>> On Mon, Sep 19, 2016 at 10:53 AM, Tommaso Teofili 
mailto:tommaso.teof...@gmail.com>> wrote:
>> Il giorno sab 17 set 2016 alle ore 15:23 Matt Post mailto:p...@cs.jhu.edu>> ha
>> scritto:
>> 
>>> I'll ask Hieu; I don't anticipate any problems. One potential problem is
>>> that that models occupy about 15--20 GB; do you think Jenkins would host
>>> this?
>>> 
>> 
>> I'm not sure, can such models be downloaded and pruned at runtime, or do
>> they need to exist on the Jenkins machine ?
>> 
>> 
>>> 
>>> (ru-en grammars still packing, results will probably not be in until 
much
>>> later today)
>>> 
>>> matt
>>> 
>>> 
 On Sep 17, 2016, at 3:19 PM, Tommaso Teofili 
mailto:tommaso.teof...@gmail.com>>
>>> wrote:
 
 Hi Matt,
 
 I think it'd be really valuable if we could be able to repeat the same
 tests (given parallel corpus is available) in the future, any chance 
you
 can share script / code to do that ? We may even consider adding a
>>> Jenkins
 job dedicated to continuously monitor performances as we work on Joshua
 master branch.
 
 WDYT?
 
 Anyway thanks for sharing the very interesting comparisons.
 Regards,
 Tommaso
 
 Il giorno sab 17 set 2016 alle ore 12:29 Matt Post mailto:p...@cs.jhu.edu>> ha
 scritto:
 
> Ugh, I think the mailing list deleted the attachment. Here is an 
attempt
> around our censors:
> 
> 
https://www.dropbox.com/s/80up63reu4q809y/ar-en-joshua-moses2.png?d

[jira] [Commented] (JOSHUA-312) Alignment is never cached

2016-09-22 Thread Matt Post (JIRA) 

[ 
https://issues.apache.org/jira/browse/JOSHUA-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513677#comment-15513677
 ] 

Matt Post commented on JOSHUA-312:
--

Hi Lewis — I don't understand. The individual steps of computing the alignment 
are all cached. If alignment completes, it will skip them. I use this all the 
time. Can you be more specific?

> Alignment is never cached
> -
>
> Key: JOSHUA-312
> URL: https://issues.apache.org/jira/browse/JOSHUA-312
> Project: Joshua
>  Issue Type: Improvement
>  Components: alignment
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.2
>
>
> Say if a pipeline fails after alignment. The alignment result is never cached 
> and it becomes necessary to undertake alignment... again!
> We should investigate the process for caching alignments as it would really 
> speed up rerunning end-to-end pipelines for large input datasets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: moses2 vs. joshua

2016-09-22 Thread Matt Post 
Hi folks,

I have finished the comparison. Here you can find graphs for ar-en and ru-en. 
The ground-up rewrite of Moses is 
about 2x–3x faster than Joshua.

http://imgur.com/a/FcIbW

One implication (untested) is that we are likely as fast as or faster than 
Moses.

We could brainstorm things to do to close this gap. I'd be much happier with 2x 
or even 1.5x than with 3x, and I bet we could narrow this down. But I'd like to 
get the 6.1 release out of the way, first, so I'm pushing this off to next 
month. Sound cool?

matt


> On Sep 19, 2016, at 6:26 AM, Matt Post  wrote:
> 
> I can't believe I did this, but I mis-colored one of the hiero lines, and the 
> Numbers legend doesn't show the line type. If you reload the dropbox file, 
> it's fixed now. The difference is about 3x for both. Here's the table.
> 
> Threads
> Joshua
> Moses2
> Joshua (hiero)
> Moses2 (hiero)
> Phrase rate
> Hiero rate
> 1
> 178
> 65
> 2116
> 1137
> 2.74
> 1.86
> 2
> 109
> 42
> 1014
> 389
> 2.60
> 2.61
> 4
> 78
> 29
> 596
> 213
> 2.69
> 2.80
> 6
> 72
> 25
> 473
> 154
> 2.88
> 3.07
> 
> I'll put the models together and share them later today. This was on a 6-core 
> machine and I agree it'd be nice to test with something much higher.
> 
> matt
> 
> 
>> On Sep 19, 2016, at 5:33 AM, kellen sunderland > > wrote:
>> 
>> Do we just want to store these models somewhere temporarily?  I've got a 
>> OneDrive account and could share the models from there (as long as they're 
>> below 500GBs or so).
>> 
>> On Mon, Sep 19, 2016 at 11:32 AM, kellen sunderland 
>> mailto:kellen.sunderl...@gmail.com>> wrote:
>> Very nice results.  I think getting to within 25% of a optimized c++ decoder 
>> from a Java decoder is impressive.  Great that Hieu has put in the work to 
>> make moses2 so fast as well, that gives organizations two quite nice 
>> decoding engines to choose from, both with reasonable performance.
>> 
>> Matt: I had a question about the x axis here.  Is that number of threads?  
>> We should be scaling more or less linearly with the number of threads, is 
>> that the case here?  If you post the models somewhere I can also do a quick 
>> benchmark on a machine with a few more cores. 
>> 
>> -Kellen
>> 
>> 
>> On Mon, Sep 19, 2016 at 10:53 AM, Tommaso Teofili > > wrote:
>> Il giorno sab 17 set 2016 alle ore 15:23 Matt Post > > ha
>> scritto:
>> 
>>> I'll ask Hieu; I don't anticipate any problems. One potential problem is
>>> that that models occupy about 15--20 GB; do you think Jenkins would host
>>> this?
>>> 
>> 
>> I'm not sure, can such models be downloaded and pruned at runtime, or do
>> they need to exist on the Jenkins machine ?
>> 
>> 
>>> 
>>> (ru-en grammars still packing, results will probably not be in until much
>>> later today)
>>> 
>>> matt
>>> 
>>> 
 On Sep 17, 2016, at 3:19 PM, Tommaso Teofili >>> >
>>> wrote:
 
 Hi Matt,
 
 I think it'd be really valuable if we could be able to repeat the same
 tests (given parallel corpus is available) in the future, any chance you
 can share script / code to do that ? We may even consider adding a
>>> Jenkins
 job dedicated to continuously monitor performances as we work on Joshua
 master branch.
 
 WDYT?
 
 Anyway thanks for sharing the very interesting comparisons.
 Regards,
 Tommaso
 
 Il giorno sab 17 set 2016 alle ore 12:29 Matt Post >>> > ha
 scritto:
 
> Ugh, I think the mailing list deleted the attachment. Here is an attempt
> around our censors:
> 
> https://www.dropbox.com/s/80up63reu4q809y/ar-en-joshua-moses2.png?dl=0 
> 
> 
> 
>> On Sep 17, 2016, at 12:21 PM, Matt Post > > wrote:
>> 
>> Hi everyone,
>> 
>> One thing we did this week at MT Marathon was a speed comparison of
> Joshua 6.1 (release candidate) with Moses2, which is a ground-up
>>> rewrite of
> Moses designed for speed (see the attached paper). Moses2 is 4–6x faster
> than Moses phrase-based, and 100x (!) faster than Moses hiero.
>> 
>> I tested using two moderate-to-large sized datasets that Hieu Hoang
> (CC'd) provided me with: ar-en and ru-en. Timing results are from 10,000
> sentences in each corpus. The average ar-en sentence length is 7.5, and
>>> for
> ru-en is 28. I only ran one test for each language, so there could be
>>> some
> variance if I averaged, but I think the results look pretty consistent.
>>> The
> timing is end-to-end (including model load times, which Moses2 tends to
>>> be
> a bit faster at).
>> 
>> Note also that Joshua does not have lexicalized distortion, while
>>> Moses2
> does. This means the BLEU scores are a bit lower for Joshua: 62.85
>>> versus

Re: [GitHub] incubator-joshua issue #65: This PR is a first attempt to minimize ngram arr...

2016-09-22 Thread Matt Post 
Sounds good to me.


> On Sep 21, 2016, at 9:14 AM, KellenSunderland  wrote:
> 
> Github user KellenSunderland commented on the issue:
> 
>https://github.com/apache/incubator-joshua/pull/65
> 
>Well in that case I can open a PR without the DirectBuffer changes 
> included.  They're pretty complicated and if they don't improve performance 
> it's probably better to leave them out.
> 
> 
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---