Hi Andy, Yep no issues with the runs in cluster in both the patches. Only difference is the Java version compatibility.
- https://github.com/apache/mahout/pull/247 - Lucene 6.1.0 - Works with Java 8 and above - The Hadoop cluster should also be running on Java 8 and above - https://github.com/apache/mahout/pull/248 - Lucene 5.5.2 - Works with Java 7 and above - The Hadoop cluster should also be running on Java 7 and above Thanks, Raviteja On Mon, Aug 8, 2016 at 2:06 PM, Andrew Palumbo <ap....@outlook.com> wrote: > Thanks Raviteja, > > > Someone will review the PR shortly. > > > And these ran w/o issue for you in cluster mode, correct? > > > Andy > > ________________________________ > From: Raviteja Lokineni <raviteja.lokin...@gmail.com> > Sent: Sunday, August 7, 2016 9:01:08 PM > To: mahout > Subject: Re: MAHOUT-1876 - Lucene compatibility > > Submitted another PR with Lucene 5.5.2 and Java 7 compatibility. Based on > the devs preference we can choose one of patches. > > https://github.com/apache/mahout/pull/248 > > I did all the necessary tests specified above and all are successful. > > Thanks, > Raviteja > > On Sat, Aug 6, 2016 at 7:20 PM, Raviteja Lokineni < > raviteja.lokin...@gmail.com> wrote: > > > *correction: wasn't working with Java 7 > > > > On Sat, Aug 6, 2016 at 10:17 PM, Raviteja Lokineni < > > raviteja.lokin...@gmail.com> wrote: > > > >> Hi Andy, > >> > >> I ran the following tests as you have specified: > >> > >> - classify-wikipedia.sh > >> - Option 2 > >> - cluster-reuters.sh > >> - Option 1,2 > >> - classify-20newsgroups.sh > >> - Option 1 > >> > >> All these examples *ran successfully* on a cloudera quickstart vm 5.7. I > >> had to change the cluster JVM to 1.8 to make it work otherwise lucene > was > >> failing with incompatible class major/minor version error (because > lucene > >> 6.1.0 was built for JVM 1.8). > >> > >> On seeing that this patch wasn't working with Java 8, I was like why, > >> why, why? > >> > >> Thanks, > >> Raviteja > >> > >> On Sat, Aug 6, 2016 at 5:15 PM, Raviteja Lokineni < > >> raviteja.lokin...@gmail.com> wrote: > >> > >>> I will let you know by tomorrow. Will run them now. > >>> > >>> On Aug 6, 2016 5:13 PM, "Andrew Palumbo" <ap....@outlook.com> wrote: > >>> > >>>> We will likely move to Java 8 at some point of course, but I > personally > >>>> would not be inclined to enforce it right now as most of our current > new > >>>> work is Scala-based, and this (the lucene dep.) is only used in > legacy > >>>> components. Admittedly though, one useful legacy component. Were > you > >>>> able to get the examples to run in pseudo-cluster mode with lucene 6? > >>>> > >>>> > >>>> Thanks, > >>>> > >>>> > >>>> Andy > >>>> > >>>> > >>>> ________________________________ > >>>> From: Andrew Palumbo <ap....@outlook.com> > >>>> Sent: Saturday, August 6, 2016 5:03:45 PM > >>>> To: dev@mahout.apache.org > >>>> Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > >>>> Thank you Raviteja, this is something that we will have to discuss. > >>>> > >>>> ________________________________ > >>>> From: Raviteja Lokineni <raviteja.lokin...@gmail.com> > >>>> Sent: Friday, August 5, 2016 11:41:09 PM > >>>> To: mahout > >>>> Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > >>>> Guys, found an issue lucene 6.x is compatible only with Java 8. What's > >>>> the > >>>> plan for mahout compatibility? Do you guys want to call in a vote for > >>>> Java > >>>> compatibility? > >>>> > >>>> On Aug 5, 2016 4:58 PM, "Andrew Palumbo" <ap....@outlook.com> wrote: > >>>> > >>>> > Hi Raviteja, > >>>> > > >>>> > > >>>> > Since this upgrade affects the entire Mahout MapReduce text > processing > >>>> > pipeline it is important to make sure that it is working in the end > >>>> to end > >>>> > examples. > >>>> > > >>>> > > >>>> > Could you please set up a Hadoop 2.4.1 pseudo cluster and run > through > >>>> the > >>>> > previously mentioned examples? > >>>> > > >>>> > > >>>> > The instructions are here (this is from 2.7.1 but should be the same > >>>> for > >>>> > 2.4.1) : > >>>> > > >>>> > > >>>> > <https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/h > >>>> adoop-common/ > >>>> > SingleCluster.html>https://hadoop.apache.org/docs/r2.7.2/ > >>>> > hadoop-project-dist/hadoop-common/SingleCluster.html# > >>>> > Pseudo-Distributed_Operation > >>>> > > >>>> > > >>>> > Thanks very much, > >>>> > > >>>> > > >>>> > Andy > >>>> > > >>>> > > >>>> > > >>>> > ________________________________ > >>>> > From: Andrew Palumbo <ap....@outlook.com> > >>>> > Sent: Friday, August 5, 2016 2:38 PM > >>>> > To: dev@mahout.apache.org > >>>> > Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > > >>>> > Ahh- yes I think we started removing MAHOUT_LOCAL capability I see > the > >>>> > check for MAHOUT_LOCAL was removed in this commit: > >>>> > > >>>> > > >>>> > https://github.com/apache/mahout/commit/daad3a4ce618cbd05be4 > >>>> 68c4ce6e45 > >>>> > 1618f3a028 > >>>> > [https://avatars3.githubusercontent.com/u/692523?v=3&s=200 > ]<https:// > >>>> > github.com/apache/mahout/commit/daad3a4ce618cbd05be468c4ce6e > >>>> 451618f3a028> > >>>> > > >>>> > MAHOUT-1665: Update hadoop commands in example scripts (akm) closes > >>>> a… · > >>>> > apache/mahout@daad3a4<https://github.com/apache/mahout/commit/ > >>>> > daad3a4ce618cbd05be468c4ce6e451618f3a028> > >>>> > github.com > >>>> > …pache/mahout#98 > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > So it would make sense that you are seeing that Error in local mode. > >>>> > > >>>> > ________________________________ > >>>> > From: Raviteja Lokineni <raviteja.lokin...@gmail.com> > >>>> > Sent: Friday, August 5, 2016 2:28:08 PM > >>>> > To: mahout > >>>> > Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > > >>>> > Nope in a Linux environment. > >>>> > > >>>> > On Aug 5, 2016 2:21 PM, "Suneel Marthi" <smar...@apache.org> wrote: > >>>> > > >>>> > > r u running this on windows prompt or in Cygwin. > >>>> > > > >>>> > > Suggest use Cygwin. > >>>> > > > >>>> > > On Fri, Aug 5, 2016 at 2:15 PM, Raviteja Lokineni < > >>>> > > raviteja.lokin...@gmail.com> wrote: > >>>> > > > >>>> > > > This is what I get. > >>>> > > > > >>>> > > > $ ./classify-20newsgroups.sh > >>>> > > > /home/lok268/projects/mahout/examples/bin/set-dfs-commands.sh: > >>>> line > >>>> > > > 36: /bin/hadoop: No such file or directory > >>>> > > > /home/lok268/projects/mahout/examples/bin/set-dfs-commands.sh: > >>>> line > >>>> > > > 38: [: too many arguments > >>>> > > > /home/lok268/projects/mahout/examples/bin/set-dfs-commands.sh: > >>>> line > >>>> > > > 43: [: -eq: unary operator expected > >>>> > > > Can't determine Hadoop version. > >>>> > > > > >>>> > > > > >>>> > > > On Fri, Aug 5, 2016 at 2:08 PM, Suneel Marthi < > smar...@apache.org > >>>> > > >>>> > > wrote: > >>>> > > > > >>>> > > > > u don't need a hadoop cluster for that, > >>>> > > > > > >>>> > > > > set MAHOUT_LOCAL=true > >>>> > > > > and u shuld be able to run locally > >>>> > > > > > >>>> > > > > On Fri, Aug 5, 2016 at 1:57 PM, Raviteja Lokineni < > >>>> > > > > raviteja.lokin...@gmail.com> wrote: > >>>> > > > > > >>>> > > > > > Hi Andrew, > >>>> > > > > > > >>>> > > > > > Looks like the examples don't seem to work unless on a > hadoop > >>>> > > cluster. > >>>> > > > > If I > >>>> > > > > > get some time I will download a cloudera quickstart vm and > >>>> test it > >>>> > > out. > >>>> > > > > > > >>>> > > > > > Thanks, > >>>> > > > > > Raviteja > >>>> > > > > > > >>>> > > > > > On Fri, Aug 5, 2016 at 12:53 PM, Andrew Palumbo < > >>>> > ap....@outlook.com> > >>>> > > > > > wrote: > >>>> > > > > > > >>>> > > > > > > Thanks again Raviteja, > >>>> > > > > > > Tests pass in my Linux env as well. > >>>> > > > > > > > >>>> > > > > > > FYI, if the windows script has not yet been officially > >>>> deprecated > >>>> > > it > >>>> > > > > > > should be soon. > >>>> > > > > > > > >>>> > > > > > > As Suneel said, someone will merge it over the weekend. > In > >>>> the > >>>> > > > > meantime > >>>> > > > > > > it would good to ensure that some of the examples are > >>>> working in > >>>> > > the > >>>> > > > > > > $MAHOUT_HOME/examples/bin dir. Could you try running > >>>> > > > > > > classify-wikipedia.sh option (2), cluster-reuters.sh > >>>> option (1) > >>>> > pr > >>>> > > > (2) > >>>> > > > > > and > >>>> > > > > > > classify-20newsgroups.sh option 1 in (pseudo)cluster mode > if > >>>> > > > possible? > >>>> > > > > > > > >>>> > > > > > > This would to ensure that seq2sparse is working correctly > >>>> which > >>>> > > > relies > >>>> > > > > > > heavily on lucene. > >>>> > > > > > > > >>>> > > > > > > Thanks again for the great contribution. > >>>> > > > > > > > >>>> > > > > > > Andy > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > -------- Original message -------- > >>>> > > > > > > From: Raviteja Lokineni <raviteja.lokin...@gmail.com> > >>>> > > > > > > Date: 08/05/2016 12:42 PM (GMT-05:00) > >>>> > > > > > > To: mahout <dev@mahout.apache.org> > >>>> > > > > > > Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > > > > > > > >>>> > > > > > > Just a FYI, all the tests are successful on windows too ;) > >>>> > > > > > > > >>>> > > > > > > On Fri, Aug 5, 2016 at 12:18 PM, Andrew Palumbo < > >>>> > > ap....@outlook.com> > >>>> > > > > > > wrote: > >>>> > > > > > > > >>>> > > > > > > > +1 > >>>> > > > > > > > > >>>> > > > > > > > ________________________________ > >>>> > > > > > > > From: Raviteja Lokineni <raviteja.lokin...@gmail.com> > >>>> > > > > > > > Sent: Friday, August 5, 2016 12:14:24 PM > >>>> > > > > > > > To: mahout > >>>> > > > > > > > Subject: Re: MAHOUT-1876 - Lucene compatibility > >>>> > > > > > > > > >>>> > > > > > > > Yay! for the heads up on merging. > >>>> > > > > > > > > >>>> > > > > > > > FYI, I take back my word on failure on windows though. I > >>>> had to > >>>> > > > > include > >>>> > > > > > > the > >>>> > > > > > > > hadoop.dll file on PATH. Tests are running (I am running > >>>> it > >>>> > just > >>>> > > to > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > -- > >>>> > > > > > *Raviteja Lokineni* | Business Intelligence Developer > >>>> > > > > > TD Ameritrade > >>>> > > > > > > >>>> > > > > > E: raviteja.lokin...@gmail.com > >>>> > > > > > > >>>> > > > > > [image: View Raviteja Lokineni's profile on LinkedIn] > >>>> > > > > > <http://in.linkedin.com/in/ravitejalokineni> > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > -- > >>>> > > > *Raviteja Lokineni* | Business Intelligence Developer > >>>> > > > TD Ameritrade > >>>> > > > > >>>> > > > E: raviteja.lokin...@gmail.com > >>>> > > > > >>>> > > > [image: View Raviteja Lokineni's profile on LinkedIn] > >>>> > > > <http://in.linkedin.com/in/ravitejalokineni> > >>>> > > > > >>>> > > > >>>> > > >>>> > >>> > >> > >> > >> -- > >> *Raviteja Lokineni* | Business Intelligence Developer > >> TD Ameritrade > >> > >> E: raviteja.lokin...@gmail.com > >> > >> [image: View Raviteja Lokineni's profile on LinkedIn] > >> <http://in.linkedin.com/in/ravitejalokineni> > >> > >> > > > > > > -- > > *Raviteja Lokineni* | Business Intelligence Developer > > TD Ameritrade > > > > E: raviteja.lokin...@gmail.com > > > > [image: View Raviteja Lokineni's profile on LinkedIn] > > <http://in.linkedin.com/in/ravitejalokineni> > > > > > > > -- > *Raviteja Lokineni* | Business Intelligence Developer > TD Ameritrade > > E: raviteja.lokin...@gmail.com > > [image: View Raviteja Lokineni's profile on LinkedIn] > <http://in.linkedin.com/in/ravitejalokineni> > -- *Raviteja Lokineni* | Business Intelligence Developer TD Ameritrade E: raviteja.lokin...@gmail.com [image: View Raviteja Lokineni's profile on LinkedIn] <http://in.linkedin.com/in/ravitejalokineni>
