Hi , With both PHOENIX-1184 , PHOENIX-1183 tickets fixed, I happened to do a quick test of Pig scripts on a CDH 5.1.0 backed cluster and things are looking good. Below are the steps I followed, a) Downloaded the binaries from https://dist.apache.org/repos/dist/dev/phoenix/phoenix-4.1.0-rc1/bin/ b) Copied phoenix-4.1.0-server-hadoop2.jar to HBase Region Server lib path and restarted. c) Copied phoenix-4.1.0-client-hadoop2.jar and phoenix-pig-4.1.0-hadoop2.jar on to the gateway node where I planned to run my Pig scripts. d) Registered the following jars within the Pig script zookeeper.jar hbase-hadoop2-compat.jar hbase-client.jar hbase-protocol-0.98.1-cdh5.1.0.jar phoenix-4.1.0-client-hadoop2.jar phoenix-pig-4.1.0-hadoop2.jar
Regards Ravi On Tue, Aug 19, 2014 at 5:43 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > I agree the vendor should resolve these issues. Hortonworks has already > included Phoenix in HDP. Cloudera is behind the curve here. I'm told > they'll include Phoenix when they feel they can support it well. > > That being said, wouldn't adding CDH/HDP options in pom.xml make the > project easier to use, and result in more people trying to use CDH/HDP with > Phoenix (and more peiople using Phoenix in general), which would bring up > bugs like the ones here? Ideally the vendors would fix these JIRAs. That > would seem to be a good thing. > > ᐧ > > > On Tue, Aug 19, 2014 at 5:34 PM, Andrew Purtell <apurt...@apache.org> > wrote: > >> Maybe "pick on" didn't get close enough to what I was after. >> >> > Maybe this is something I can fix. If I were to add the >> cloudera/hortonworks maven repos, and then add some supported options for >> hadoop beyond 1/2, that would pretty much do it, right? >> >> I doubt it, because v4 and master branches probably won't compile against >> either, certainly not against CDH 5.1, their HBase releases are out of step >> and stale with respect to the latest Apache HBase 0.98 and Apache Phoenix 4 >> releases. >> >> Getting back to my point, it's unfair in my opinion to expect the >> upstream volunteer Apache projects to track all of the commercial options >> and the vagaries of their arbitrary code freezes and "curated" additional >> patches. It's unfair to expect Salesforce to fund such an effort, unless >> Salesforce has somehow gone into the Hadoop distribution business. >> Certainly I am not speaking on behalf of Salesforce or anyone else here. On >> the other hand, I think it would be totally reasonable to request your >> favorite vendor address Phoenix related issues with *their* derivative >> distributions. >> >> >> >> On Tue, Aug 19, 2014 at 3:53 PM, Russell Jurney <russell.jur...@gmail.com >> > wrote: >> >>> Maybe this is something I can fix. If I were to add the >>> cloudera/hortonworks maven repos, and then add some supported options for >>> hadoop beyond 1/2, that would pretty much do it, right? >>> ᐧ >>> >>> >>> On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <jesse.k.ya...@gmail.com> >>> wrote: >>> >>>> FWIW internally at Salesforce we also patch the HBase and Hadoop poms >>>> to support our own internal 'light forks'. Its really not a big deal to >>>> manage - a couple of jenkins jobs (one to automate, one to track open >>>> source changes and ensure your patch(es) still work, etc) and you are good >>>> to go. >>>> >>>> I imagine this is also what various distributors are doing for their >>>> forks as well. >>>> >>>> ------------------- >>>> Jesse Yates >>>> @jesse_yates >>>> jyates.github.com >>>> >>>> >>>> On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney < >>>> russell.jur...@gmail.com> wrote: >>>> >>>>> First of all, I apologize if you feel like I was picking on you. I was >>>>> not trying to do that. >>>>> >>>>> My understanding is that Salesforce pays people to work on Phoenix. Is >>>>> that not the case? I'm hoping one of them will add spark-like support for >>>>> CDH and HDP to advance the project. >>>>> >>>>> And I don't mention the POM thing to pick on someone. The majority of >>>>> HBase users are not going to be able to use Phoenix because they run a >>>>> commercial distribution of Hadoop and aren't pom wizards. That seems kind >>>>> of important for the well being of the project. >>>>> ᐧ >>>>> >>>>> >>>>> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurt...@apache.org> >>>>> wrote: >>>>> >>>>>> I don't think an Apache project should spend precious bandwidth >>>>>> tracking the various and sundry redistributors of Apache ecosystem >>>>>> projects. This is putting the cart before the horse. The horse is the >>>>>> Apache upstream projects. The cart is the commercial distributions >>>>>> leveraging the Apache ecosystem for profit. Spark is not a good example, >>>>>> it >>>>>> is supported by a commercial concern, Databricks. What commercial company >>>>>> supports Phoenix? Why do you think it is appropriate to pick on >>>>>> volunteers >>>>>> because editing POM files is too much work? >>>>>> >>>>>> >>>>>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney < >>>>>> russell.jur...@gmail.com> wrote: >>>>>> >>>>>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185 >>>>>>> because requiring users to hand-edit the pom.xml just to build against >>>>>>> CDH >>>>>>> and HDP is nuts. >>>>>>> ᐧ >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney < >>>>>>> russell.jur...@gmail.com> wrote: >>>>>>> >>>>>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 >>>>>>>> and https://issues.apache.org/jira/browse/PHOENIX-1184 >>>>>>>> >>>>>>>> I hope this can be resolved soon, we are pretty screwed right now. >>>>>>>> ᐧ >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates < >>>>>>>> jesse.k.ya...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Yup, that looks like an issue to me :-/ >>>>>>>>> >>>>>>>>> ------------------- >>>>>>>>> Jesse Yates >>>>>>>>> @jesse_yates >>>>>>>>> jyates.github.com >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney < >>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Running against any version would be ok, but it does not work. I >>>>>>>>>> get this error: >>>>>>>>>> >>>>>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error >>>>>>>>>> running child : java.lang.IncompatibleClassChangeError: Found >>>>>>>>>> interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class >>>>>>>>>> was expected >>>>>>>>>> >>>>>>>>>> ᐧ >>>>>>>>>> I went back and checked out the Phoenix commit before the >>>>>>>>>> pom.xml was upgraded to 0.98.4 >>>>>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>, >>>>>>>>>> and I get the same error. >>>>>>>>>> >>>>>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as >>>>>>>>>> CDH 5 is probably the most popular installation there is? >>>>>>>>>> >>>>>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor < >>>>>>>>>> jamestay...@apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> The dependencies on HBase 0.98.4 are *compile time* >>>>>>>>>>> dependencies. Is it necessary for you to compile against CDH 5.1 or >>>>>>>>>>> just >>>>>>>>>>> run against it? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tuesday, August 19, 2014, Russell Jurney < >>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How >>>>>>>>>>>> can this be fixed? I'm not sure what to do. We're in limbo on our >>>>>>>>>>>> new >>>>>>>>>>>> cluster now. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran < >>>>>>>>>>>> maghamraviki...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Russel, >>>>>>>>>>>>> Apparently, Phoenix 4.0.0 leverages few API methods of >>>>>>>>>>>>> HBase 0.98.4 v which aren't present within 0.98.1 that comes >>>>>>>>>>>>> with CDH 5.1 >>>>>>>>>>>>> . That's the primary cause for the build issues. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Ravi >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney < >>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Talking to myself, but hopefully creating good docs. >>>>>>>>>>>>>> Replacing the previous hadoop version with one I found here: >>>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, >>>>>>>>>>>>>> 2.3.0-mr1-cdh5.1.0, >>>>>>>>>>>>>> makes things get a little further. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can't get past some build errors, however. Has anyone done >>>>>>>>>>>>>> this before me who can help? >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>>>> cannot find symbol >>>>>>>>>>>>>> >>>>>>>>>>>>>> symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>>>> >>>>>>>>>>>>>> location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>>>>>>>>>>>> (default-compile) on project phoenix-core: Compilation failure >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>>>> cannot find symbol >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] -> [Help 1] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney < >>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, so it is clear to me what I have to do. I have to edit >>>>>>>>>>>>>>> my pom.xml to point at CDH 5.1, which translates into: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Add the cloudera repo: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <repository> >>>>>>>>>>>>>>> <id>cloudera</id> >>>>>>>>>>>>>>> <url> >>>>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/ >>>>>>>>>>>>>>> </url> >>>>>>>>>>>>>>> </repository> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then change the hadoop and hbase versions: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <!-- Hadoop Versions --> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version> >>>>>>>>>>>>>>> <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version> >>>>>>>>>>>>>>> <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However, I get this error when I build, which tells me there >>>>>>>>>>>>>>> is more complex POM surgery required. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core: >>>>>>>>>>>>>>> Could not resolve dependencies for project >>>>>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The >>>>>>>>>>>>>>> following >>>>>>>>>>>>>>> artifacts could not be resolved: >>>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, >>>>>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not >>>>>>>>>>>>>>> find artifact >>>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache >>>>>>>>>>>>>>> release ( >>>>>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/) >>>>>>>>>>>>>>> -> [Help 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Beyond changing the versions, I do not know how to fix this. >>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In general, is it possible to actually handle different CDH >>>>>>>>>>>>>>> versions in this project? One shouldn't have to do pom surgery >>>>>>>>>>>>>>> to build >>>>>>>>>>>>>>> Phoenix for the most common platform. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney < >>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When I try to store data into Phoenix from Pig, I get this >>>>>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyone know how to resolve this issue? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO >>>>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: >>>>>>>>>>>>>>>> Current split being processed >>>>>>>>>>>>>>>> hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845 >>>>>>>>>>>>>>>> 2014-08-18 >>>>>>>>>>>>>>>> <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18> >>>>>>>>>>>>>>>> 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>>>>>>>>>>>>> Initializing logs' truncater with mapRetainSize=-1 and >>>>>>>>>>>>>>>> reduceRetainSize=-1 >>>>>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: >>>>>>>>>>>>>>>> Error running child : java.lang.IncompatibleClassChangeError: >>>>>>>>>>>>>>>> Found interface >>>>>>>>>>>>>>>> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was >>>>>>>>>>>>>>>> expected >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >>>>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>>>>>> Method) >>>>>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) >>>>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>> datasyndrome.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>> datasyndrome.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>> datasyndrome.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> >>>>>> - Andy >>>>>> >>>>>> Problems worthy of attack prove their worth by hitting back. - Piet >>>>>> Hein (via Tom White) >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>> datasyndrome.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome >>> .com >>> >> >> >> >> -- >> Best regards, >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> > > > > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome. > com >
