In the end I was able to recompile Nifi to work with Hive 1.1 by providing the appropriate profile and hive version (in my case -Pcloudera -Dhadoop.version=2.6.0-cdh5.5.2 -Dhive.version=1.1.0-cdh5.5.2 ) as well as modifying a couple of files related to Hive ORC support: the change I did was quite brutal as I basically removed all the code which was generating compilation errors but since it involved only 2 files related to hive-ORC (that we don't use), it was way faster than trying to fix all the dependencies (I think that a proper fix would not be that much complicated anyway). With these changes in place I was able to connect to an Hive Kerberos-secured server and interact with Hive tables without any issue (but, as you can imagine, I didn't test any ORC table).
Thanks for the support, Yari On 9 September 2016 at 15:26, Yari Marchetti <yari.marche...@buongiorno.com> wrote: > I suspected the same, and that's what I've been trying to accomplish so > far (with little luck). Thanks for confirming, Matt. > I'll try to see if we can upgrade our Hive version to Hive 1.2.1 in the > mean time. > > thanks, > Yari > > > > > On 9 September 2016 at 15:10, Matt Burgess <mattyb...@gmail.com> wrote: > >> Yari, >> >> NiFi was coded and built against Apache Hive 1.2.1. Some of the API, >> files, and folders had changed between Hive 1.1.0 and Hive 1.2.1, such >> as the org.apache.hadoop.hive.ql.io.filters package being added in >> 1.2.0. Bringing in Apache ORC won't work for the same reason, it was >> split from Hive after 1.2.0. In fact in Hive 1.1.0 I don't think there >> is even a hive-orc JAR, it is just part of another Hive JAR. Some >> classes didn't get moved to Apache ORC from Hive until 2.0, and some >> got moved to different packages (hence your missing classes errors >> below). >> >> I think the bottom line is that you'd probably have to change the >> source code in order to get NiFi to build against an older Hive >> version. >> >> Regards, >> Matt >> >> On Fri, Sep 9, 2016 at 8:50 AM, Yari Marchetti >> <yari.marche...@buongiorno.com> wrote: >> > Thanks, Andre. This is the command I'm using: >> > >> > mvn install -DskipTests -Pcloudera -Dhadoop.version=2.6.0-cdh5.5.2 >> > -Dhive.version=1.1.0-cdh5.5.2 >> > >> > Adding as dependency: >> > >> > <dependency> >> > <groupId>org.apache.orc</groupId> >> > <artifactId>orc-core</artifactId> >> > <version>1.1.0</version> >> > </dependency> >> > >> > Yari >> > >> > >> > On 9 September 2016 at 13:23, Andre <andre-li...@fucs.org> wrote: >> >> >> >> Yari, >> >> >> >> Can you please confirm what command line are you using to compile? >> >> >> >> Happy yo try to reproduce. >> >> >> >> Cheers >> >> >> >> On Fri, Sep 9, 2016 at 8:49 PM, Yari Marchetti >> >> <yari.marche...@buongiorno.com> wrote: >> >>> >> >>> Hi Peter/Andre, >> >>> I cherrypicked the commit and added the dependency on: >> >>> >> >>> <dependency> >> >>> <groupId>org.apache.orc</groupId> >> >>> <artifactId>orc-core</artifactId> >> >>> <version>1.1.0</version> >> >>> </dependency> >> >>> >> >>> (tried also with version 1.2.0) but stil no luck, as I keep on getting >> >>> compilation errors: >> >>> >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[200,20] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: variable OrcUtils >> >>> [ERROR] location: class OrcFlowFileWriter >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[203,20] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: variable OrcUtils >> >>> [ERROR] location: class OrcFlowFileWriter >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[313,45] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: method loadClass(String) >> >>> [ERROR] location: class JavaUtils >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[494,21] >> >>> error: an enum switch case label must be the unqualified name of an >> >>> enumeration constant >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[689,43] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: variable BloomFilter >> >>> [ERROR] location: class OrcProto >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[690,43] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: variable BloomFilterIndex >> >>> [ERROR] location: class OrcProto >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[691,87] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: variable BLOOM_FILTER >> >>> [ERROR] location: class Kind >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[692,34] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: class BloomFilterIO >> >>> [ERROR] location: class TreeWriter >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[726,52] >> >>> error: incompatible types: PositionedOutputStream cannot be converted >> to >> >>> OutStream >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[728,50] >> >>> error: incompatible types: PositionedOutputStream cannot be converted >> to >> >>> OutStream >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowF >> ileWriter.java:[804,19] >> >>> error: cannot find symbol >> >>> [ERROR] symbol: method setWriterTimezone(String) >> >>> [ERROR] location: variable builder of type Builder >> >>> [ERROR] >> >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors >> /src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcU >> tils.java:[450,32] >> >>> error: cannot find symbol >> >>> >> >>> do you have any idea? >> >>> >> >>> Thanks, >> >>> Yari >> >>> >> >>> On 8 September 2016 at 20:42, Peter Wicks (pwicks) <pwi...@micron.com >> > >> >>> wrote: >> >>>> >> >>>> Also, ORC File support was pulled out into its own library on the >> HIVE >> >>>> side. >> >>>> >> >>>> If you are willing to compile and run your own version you might >> need to >> >>>> include orc-core as a MVN dependency: >> >>>> https://mvnrepository.com/artifact/org.apache.orc/orc-core/1.2.0. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> From: Andre [mailto:andre-li...@fucs.org] >> >>>> Sent: Thursday, September 08, 2016 4:51 AM >> >>>> To: users@nifi.apache.org >> >>>> Subject: Re: Nifi 1.0.0 compatibility with Hive 1.1.0 >> >>>> >> >>>> >> >>>> >> >>>> Yari, >> >>>> >> >>>> >> >>>> >> >>>> Is there any chance you can cherry pick commit >> >>>> 80224e3e5ed7ee7b09c4985a920a7fa393bff26c and try again? >> >>>> >> >>>> >> >>>> >> >>>> Post 1.0.0 there have been some changes to streamline compilation >> using >> >>>> vendor provided libraries. >> >>>> >> >>>> >> >>>> >> >>>> Cheers >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Sep 8, 2016 at 8:44 PM, Yari Marchetti >> >>>> <yari.marche...@buongiorno.com> wrote: >> >>>> >> >>>> Hello, >> >>>> >> >>>> I'd like to use Nifi 1.0.0 with Hive 1.1.0 (on CDH 5.5.2) but after >> some >> >>>> investigation I realised that the hive-jdbc driver included in Nifi >> is >> >>>> incompatible with the Hive version we're using (1.1.0 on CDH 5.5.2) >> as I'm >> >>>> getting the error: >> >>>> >> >>>> >> >>>> >> >>>> org.apache.hive.jdbc.HiveConnection Error opening session >> >>>> >> >>>> org.apache.thrift.TApplicationException: Required field >> >>>> 'client_protocol' is unset! Struct:TOpenSessionReq(client_ >> protocol:null, >> >>>> configuration:{use:database=unifieddata}) >> >>>> >> >>>> >> >>>> >> >>>> So I just tried to recompile Nifi using the Cloudera profile 5.5.2 >> but >> >>>> compilation is failing: >> >>>> >> >>>> >> >>>> >> >>>> [ERROR] Failed to execute goal >> >>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile >> (default-compile) >> >>>> on project nifi-hive-processors: Compilation failure: Compilation >> failure: >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/NiFiOrcUtils.java:[26,43] >> >>>> error: package org.apache.hadoop.hive.ql.io.filters does not exist >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/OrcFlowFileWriter.java:[45,43] >> >>>> error: package org.apache.hadoop.hive.ql.io.filters does not exist >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/OrcFlowFileWriter.java:[643,24] >> >>>> error: cannot find symbol >> >>>> >> >>>> [ERROR] symbol: class BloomFilterIO >> >>>> >> >>>> [ERROR] location: class TreeWriter >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/OrcFlowFileWriter.java:[645,30] >> >>>> error: cannot find symbol >> >>>> >> >>>> [ERROR] symbol: class BloomFilterIndex >> >>>> >> >>>> [ERROR] location: class OrcProto >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/OrcFlowFileWriter.java:[646,30] >> >>>> error: cannot find symbol >> >>>> >> >>>> [ERROR] symbol: class BloomFilter >> >>>> >> >>>> [ERROR] location: class OrcProto >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/NiFiOrcUtils.java:[450,32] >> >>>> error: cannot find symbol >> >>>> >> >>>> [ERROR] symbol: variable BloomFilterIO >> >>>> >> >>>> [ERROR] location: class NiFiOrcUtils >> >>>> >> >>>> [ERROR] >> >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi >> -hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/ >> orc/OrcFlowFileWriter.java:[200,20] >> >>>> error: cannot find symbol >> >>>> >> >>>> [ERROR] symbol: variable OrcUtils >> >>>> >> >>>> [ERROR] location: class OrcFlowFileWriter >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> Is there any way to get Nifi to work with Hive 1.1.0 and CDH 5.5.2? >> >>>> >> >>>> >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Yari >> >>>> >> >>>> >> >>> >> >>> >> >> >> > >> > >
