FWIW internally at Salesforce we also patch the HBase and Hadoop poms to support our own internal 'light forks'. Its really not a big deal to manage - a couple of jenkins jobs (one to automate, one to track open source changes and ensure your patch(es) still work, etc) and you are good to go.
I imagine this is also what various distributors are doing for their forks as well. ------------------- Jesse Yates @jesse_yates jyates.github.com On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > First of all, I apologize if you feel like I was picking on you. I was not > trying to do that. > > My understanding is that Salesforce pays people to work on Phoenix. Is > that not the case? I'm hoping one of them will add spark-like support for > CDH and HDP to advance the project. > > And I don't mention the POM thing to pick on someone. The majority of > HBase users are not going to be able to use Phoenix because they run a > commercial distribution of Hadoop and aren't pom wizards. That seems kind > of important for the well being of the project. > ᐧ > > > On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurt...@apache.org> > wrote: > >> I don't think an Apache project should spend precious bandwidth tracking >> the various and sundry redistributors of Apache ecosystem projects. This is >> putting the cart before the horse. The horse is the Apache upstream >> projects. The cart is the commercial distributions leveraging the Apache >> ecosystem for profit. Spark is not a good example, it is supported by a >> commercial concern, Databricks. What commercial company supports Phoenix? >> Why do you think it is appropriate to pick on volunteers because editing >> POM files is too much work? >> >> >> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney <russell.jur...@gmail.com >> > wrote: >> >>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185 >>> because requiring users to hand-edit the pom.xml just to build against CDH >>> and HDP is nuts. >>> ᐧ >>> >>> >>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney < >>> russell.jur...@gmail.com> wrote: >>> >>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and >>>> https://issues.apache.org/jira/browse/PHOENIX-1184 >>>> >>>> I hope this can be resolved soon, we are pretty screwed right now. >>>> ᐧ >>>> >>>> >>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.ya...@gmail.com> >>>> wrote: >>>> >>>>> Yup, that looks like an issue to me :-/ >>>>> >>>>> ------------------- >>>>> Jesse Yates >>>>> @jesse_yates >>>>> jyates.github.com >>>>> >>>>> >>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney < >>>>> russell.jur...@gmail.com> wrote: >>>>> >>>>>> Running against any version would be ok, but it does not work. I get >>>>>> this error: >>>>>> >>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error >>>>>> running child : java.lang.IncompatibleClassChangeError: Found interface >>>>>> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected >>>>>> >>>>>> ᐧ >>>>>> I went back and checked out the Phoenix commit before the pom.xml >>>>>> was upgraded to 0.98.4 >>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>, >>>>>> and I get the same error. >>>>>> >>>>>> Not sure what to do. Stuck. I imagine most users must be too, as CDH >>>>>> 5 is probably the most popular installation there is? >>>>>> >>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor < >>>>>> jamestay...@apache.org> wrote: >>>>>> >>>>>>> The dependencies on HBase 0.98.4 are *compile time* dependencies. >>>>>>> Is it necessary for you to compile against CDH 5.1 or just run against >>>>>>> it? >>>>>>> >>>>>>> >>>>>>> On Tuesday, August 19, 2014, Russell Jurney < >>>>>>> russell.jur...@gmail.com> wrote: >>>>>>> >>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How can >>>>>>>> this be fixed? I'm not sure what to do. We're in limbo on our new >>>>>>>> cluster >>>>>>>> now. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran < >>>>>>>> maghamraviki...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Russel, >>>>>>>>> Apparently, Phoenix 4.0.0 leverages few API methods of HBase >>>>>>>>> 0.98.4 v which aren't present within 0.98.1 that comes with CDH 5.1 . >>>>>>>>> That's the primary cause for the build issues. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Ravi >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney < >>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Talking to myself, but hopefully creating good docs. Replacing >>>>>>>>>> the previous hadoop version with one I found here: >>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, >>>>>>>>>> 2.3.0-mr1-cdh5.1.0, >>>>>>>>>> makes things get a little further. >>>>>>>>>> >>>>>>>>>> I can't get past some build errors, however. Has anyone done this >>>>>>>>>> before me who can help? >>>>>>>>>> >>>>>>>>>> [ERROR] >>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>> cannot find symbol >>>>>>>>>> >>>>>>>>>> symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>> >>>>>>>>>> location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>>>>>>>> (default-compile) on project phoenix-core: Compilation failure >>>>>>>>>> >>>>>>>>>> [ERROR] >>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>> cannot find symbol >>>>>>>>>> >>>>>>>>>> [ERROR] symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>> >>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>> >>>>>>>>>> [ERROR] -> [Help 1] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney < >>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Ok, so it is clear to me what I have to do. I have to edit my >>>>>>>>>>> pom.xml to point at CDH 5.1, which translates into: >>>>>>>>>>> >>>>>>>>>>> Add the cloudera repo: >>>>>>>>>>> >>>>>>>>>>> <repository> >>>>>>>>>>> <id>cloudera</id> >>>>>>>>>>> <url> >>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/ >>>>>>>>>>> </url> >>>>>>>>>>> </repository> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Then change the hadoop and hbase versions: >>>>>>>>>>> >>>>>>>>>>> <!-- Hadoop Versions --> >>>>>>>>>>> >>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version> >>>>>>>>>>> >>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version> >>>>>>>>>>> <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version> >>>>>>>>>>> <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> However, I get this error when I build, which tells me there is >>>>>>>>>>> more complex POM surgery required. >>>>>>>>>>> >>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core: Could >>>>>>>>>>> not resolve dependencies for project >>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The following >>>>>>>>>>> artifacts could not be resolved: >>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, >>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find >>>>>>>>>>> artifact >>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache release ( >>>>>>>>>>> https://repository.apache.org/content/repositories/releases/) >>>>>>>>>>> -> [Help 1] >>>>>>>>>>> >>>>>>>>>>> Beyond changing the versions, I do not know how to fix this. Can >>>>>>>>>>> anyone help? >>>>>>>>>>> >>>>>>>>>>> In general, is it possible to actually handle different CDH >>>>>>>>>>> versions in this project? One shouldn't have to do pom surgery to >>>>>>>>>>> build >>>>>>>>>>> Phoenix for the most common platform. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney < >>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> When I try to store data into Phoenix from Pig, I get this >>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0. >>>>>>>>>>>> >>>>>>>>>>>> Anyone know how to resolve this issue? >>>>>>>>>>>> >>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO >>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: >>>>>>>>>>>> Current split being processed >>>>>>>>>>>> hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845 >>>>>>>>>>>> 2014-08-18 >>>>>>>>>>>> <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18> >>>>>>>>>>>> 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>>>>>>>>> Initializing logs' truncater with mapRetainSize=-1 and >>>>>>>>>>>> reduceRetainSize=-1 >>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: >>>>>>>>>>>> Error running child : java.lang.IncompatibleClassChangeError: >>>>>>>>>>>> Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, >>>>>>>>>>>> but class was expected >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653) >>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) >>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>> datasyndrome.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>> datasyndrome.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>> datasyndrome.com >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>> datasyndrome.com >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>> datasyndrome.com >>>> >>> >>> >>> >>> -- >>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome >>> .com >>> >> >> >> >> -- >> Best regards, >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> > > > > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome. > com >
