Maybe "pick on" didn't get close enough to what I was after. > Maybe this is something I can fix. If I were to add the cloudera/hortonworks maven repos, and then add some supported options for hadoop beyond 1/2, that would pretty much do it, right?
I doubt it, because v4 and master branches probably won't compile against either, certainly not against CDH 5.1, their HBase releases are out of step and stale with respect to the latest Apache HBase 0.98 and Apache Phoenix 4 releases. Getting back to my point, it's unfair in my opinion to expect the upstream volunteer Apache projects to track all of the commercial options and the vagaries of their arbitrary code freezes and "curated" additional patches. It's unfair to expect Salesforce to fund such an effort, unless Salesforce has somehow gone into the Hadoop distribution business. Certainly I am not speaking on behalf of Salesforce or anyone else here. On the other hand, I think it would be totally reasonable to request your favorite vendor address Phoenix related issues with *their* derivative distributions. On Tue, Aug 19, 2014 at 3:53 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > Maybe this is something I can fix. If I were to add the > cloudera/hortonworks maven repos, and then add some supported options for > hadoop beyond 1/2, that would pretty much do it, right? > ᐧ > > > On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <jesse.k.ya...@gmail.com> > wrote: > >> FWIW internally at Salesforce we also patch the HBase and Hadoop poms to >> support our own internal 'light forks'. Its really not a big deal to manage >> - a couple of jenkins jobs (one to automate, one to track open source >> changes and ensure your patch(es) still work, etc) and you are good to go. >> >> I imagine this is also what various distributors are doing for their >> forks as well. >> >> ------------------- >> Jesse Yates >> @jesse_yates >> jyates.github.com >> >> >> On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jur...@gmail.com >> > wrote: >> >>> First of all, I apologize if you feel like I was picking on you. I was >>> not trying to do that. >>> >>> My understanding is that Salesforce pays people to work on Phoenix. Is >>> that not the case? I'm hoping one of them will add spark-like support for >>> CDH and HDP to advance the project. >>> >>> And I don't mention the POM thing to pick on someone. The majority of >>> HBase users are not going to be able to use Phoenix because they run a >>> commercial distribution of Hadoop and aren't pom wizards. That seems kind >>> of important for the well being of the project. >>> ᐧ >>> >>> >>> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurt...@apache.org> >>> wrote: >>> >>>> I don't think an Apache project should spend precious bandwidth >>>> tracking the various and sundry redistributors of Apache ecosystem >>>> projects. This is putting the cart before the horse. The horse is the >>>> Apache upstream projects. The cart is the commercial distributions >>>> leveraging the Apache ecosystem for profit. Spark is not a good example, it >>>> is supported by a commercial concern, Databricks. What commercial company >>>> supports Phoenix? Why do you think it is appropriate to pick on volunteers >>>> because editing POM files is too much work? >>>> >>>> >>>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney < >>>> russell.jur...@gmail.com> wrote: >>>> >>>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185 >>>>> because requiring users to hand-edit the pom.xml just to build against CDH >>>>> and HDP is nuts. >>>>> ᐧ >>>>> >>>>> >>>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney < >>>>> russell.jur...@gmail.com> wrote: >>>>> >>>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 >>>>>> and https://issues.apache.org/jira/browse/PHOENIX-1184 >>>>>> >>>>>> I hope this can be resolved soon, we are pretty screwed right now. >>>>>> ᐧ >>>>>> >>>>>> >>>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.ya...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Yup, that looks like an issue to me :-/ >>>>>>> >>>>>>> ------------------- >>>>>>> Jesse Yates >>>>>>> @jesse_yates >>>>>>> jyates.github.com >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney < >>>>>>> russell.jur...@gmail.com> wrote: >>>>>>> >>>>>>>> Running against any version would be ok, but it does not work. I >>>>>>>> get this error: >>>>>>>> >>>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error >>>>>>>> running child : java.lang.IncompatibleClassChangeError: Found >>>>>>>> interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class >>>>>>>> was expected >>>>>>>> >>>>>>>> ᐧ >>>>>>>> I went back and checked out the Phoenix commit before the pom.xml >>>>>>>> was upgraded to 0.98.4 >>>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>, >>>>>>>> and I get the same error. >>>>>>>> >>>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as >>>>>>>> CDH 5 is probably the most popular installation there is? >>>>>>>> >>>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor < >>>>>>>> jamestay...@apache.org> wrote: >>>>>>>> >>>>>>>>> The dependencies on HBase 0.98.4 are *compile time* dependencies. >>>>>>>>> Is it necessary for you to compile against CDH 5.1 or just run >>>>>>>>> against it? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, August 19, 2014, Russell Jurney < >>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How >>>>>>>>>> can this be fixed? I'm not sure what to do. We're in limbo on our new >>>>>>>>>> cluster now. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran < >>>>>>>>>> maghamraviki...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Russel, >>>>>>>>>>> Apparently, Phoenix 4.0.0 leverages few API methods of HBase >>>>>>>>>>> 0.98.4 v which aren't present within 0.98.1 that comes with CDH >>>>>>>>>>> 5.1 . >>>>>>>>>>> That's the primary cause for the build issues. >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> Ravi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney < >>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Talking to myself, but hopefully creating good docs. Replacing >>>>>>>>>>>> the previous hadoop version with one I found here: >>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, >>>>>>>>>>>> 2.3.0-mr1-cdh5.1.0, >>>>>>>>>>>> makes things get a little further. >>>>>>>>>>>> >>>>>>>>>>>> I can't get past some build errors, however. Has anyone done >>>>>>>>>>>> this before me who can help? >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] >>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>> cannot find symbol >>>>>>>>>>>> >>>>>>>>>>>> symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>> >>>>>>>>>>>> location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>>>>>>>>>> (default-compile) on project phoenix-core: Compilation failure >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] >>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>> cannot find symbol >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] -> [Help 1] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney < >>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, so it is clear to me what I have to do. I have to edit my >>>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates into: >>>>>>>>>>>>> >>>>>>>>>>>>> Add the cloudera repo: >>>>>>>>>>>>> >>>>>>>>>>>>> <repository> >>>>>>>>>>>>> <id>cloudera</id> >>>>>>>>>>>>> <url> >>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/ >>>>>>>>>>>>> </url> >>>>>>>>>>>>> </repository> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Then change the hadoop and hbase versions: >>>>>>>>>>>>> >>>>>>>>>>>>> <!-- Hadoop Versions --> >>>>>>>>>>>>> >>>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version> >>>>>>>>>>>>> >>>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version> >>>>>>>>>>>>> <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version> >>>>>>>>>>>>> <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> However, I get this error when I build, which tells me there >>>>>>>>>>>>> is more complex POM surgery required. >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core: Could >>>>>>>>>>>>> not resolve dependencies for project >>>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The >>>>>>>>>>>>> following >>>>>>>>>>>>> artifacts could not be resolved: >>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, >>>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find >>>>>>>>>>>>> artifact >>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache >>>>>>>>>>>>> release ( >>>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/) >>>>>>>>>>>>> -> [Help 1] >>>>>>>>>>>>> >>>>>>>>>>>>> Beyond changing the versions, I do not know how to fix this. >>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>> >>>>>>>>>>>>> In general, is it possible to actually handle different CDH >>>>>>>>>>>>> versions in this project? One shouldn't have to do pom surgery to >>>>>>>>>>>>> build >>>>>>>>>>>>> Phoenix for the most common platform. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney < >>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> When I try to store data into Phoenix from Pig, I get this >>>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyone know how to resolve this issue? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO >>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: >>>>>>>>>>>>>> Current split being processed >>>>>>>>>>>>>> hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845 >>>>>>>>>>>>>> 2014-08-18 >>>>>>>>>>>>>> <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18> >>>>>>>>>>>>>> 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>>>>>>>>>>> Initializing logs' truncater with mapRetainSize=-1 and >>>>>>>>>>>>>> reduceRetainSize=-1 >>>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: >>>>>>>>>>>>>> Error running child : java.lang.IncompatibleClassChangeError: >>>>>>>>>>>>>> Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, >>>>>>>>>>>>>> but class was expected >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653) >>>>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) >>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>>>> datasyndrome.com >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>>>> datasyndrome.com >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>> datasyndrome.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>> datasyndrome.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> >>>> - Andy >>>> >>>> Problems worthy of attack prove their worth by hitting back. - Piet >>>> Hein (via Tom White) >>>> >>> >>> >>> >>> -- >>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome >>> .com >>> >> >> > > > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome. > com > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
