Sorry, sent too fast ... either way you do it, you'll need to include all of the dependencies of the Hive JDBC driver classes in a single JAR and the apply the shading the unified JAR file. This ensures that all classes that depend on Thrift 0.9 get updated to use the shaded package. For example, I'm working with the HCatalog Pig Adapter so I built a single JAR made up of:
antlr-runtime-3.4.jar commons-io-2.4.jar hive-beeline-0.11.0.jar hive-serde-0.11.0.jar jdo2-api-2.3-ec.jar protobuf-java-2.4.1.jar avro-1.7.1.jar commons-lang-2.4.jar hive-cli-0.11.0.jar hive-service-0.11.0.jar jetty-6.1.26.jar servlet-api-2.5-20081211.jar avro-mapred-1.7.1.jar commons-logging-1.0.4.jar hive-common-0.11.0.jar hive-shims-0.11.0.jar jetty-util-6.1.26.jar slf4j-api-1.6.1.jar commons-cli-1.2.jar commons-logging-api-1.0.4.jar hive-contrib-0.11.0.jar jackson-core-asl-1.8.8.jar jline-0.9.94.jar slf4j-log4j12-1.6.1.jar commons-codec-1.4.jar commons-pool-1.5.4.jar hive-exec-0.11.0.jar jackson-jaxrs-1.8.8.jar json-20090211.jar snappy-0.2.jar commons-collections-3.2.1.jar derby-10.4.2.0.jar hive-hbase-handler-0.11.0.jar jackson-mapper-asl-1.8.8.jar libfb303-0.9.0.jar ST4-4.0.4.jar commons-compress-1.4.1.jar guava-11.0.2.jar hive-hwi-0.11.0.jar jackson-xc-1.8.8.jar libthrift-0.9.0.jar tempus-fugit-1.1.jar commons-configuration-1.6.jar hcatalog-core-0.11.0.jar hive-jdbc-0.11.0.jar JavaEWAH-0.3.2.jar log4j-1.2.16.jar xz-1.0.jar commons-dbcp-1.4.jar hcatalog-pig-adapter-0.11.0.jar hive-metastore-0.11.0.jar javolution-5.5.1.jar metrics-core-2.1.2.jar zookeeper-3.4.3.jar Any class in any of these JAR files that depend on Thrift 0.9 classes will get updated at the bytecode level to refer to my shaded package - org.shaded.thrift Make sense? On Thu, Oct 10, 2013 at 6:04 PM, Timothy Potter <thelabd...@gmail.com>wrote: > I'm not sure about keep, but if you're going to use maven, go with the > shade plugin vs. jarjar ... it does the same thing but has better maven > integration - http://maven.apache.org/plugins/maven-shade-plugin/ > > > > > On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <zhangxiaoyu...@gmail.com>wrote: > >> Hi, Timothy, >> Thanks for your reply. Seems jarjar is a solution for me. I have a basic >> question to follow: >> >> I don't quite understand what is tag <keep> for ? >> >> My understand is >> 1. >> use <include> to grab the maven dependencies I want to re-package, >> 2. >> then use <pattern> and <result> in <rule> to define what class to >> re-package and what's the new name for re-packaged binary. >> 3. >> And use <keep> to indicate which class in the project will only use this >> re-packaged dependencies, in my case, it is JDBC class. Am I right ? >> >> below pom doesn't work for me, and JDBC class and other class still share >> the same thrift 0.9. Thrift 0.7 is omitted. >> >> Thanks a lot for help, >> Johnny >> >> <dependency> >> >> <groupId>org.apache.thrift</groupId> >> >> <artifactId>libthrift</artifactId> >> >> <version>0.9.0</version> >> >> </dependency> >> >> ...... >> >> ...... >> >> <plugin> >> >> <groupId>org.sonatype.plugins</groupId> >> >> <artifactId>jarjar-maven-plugin</artifactId> >> >> <executions> >> >> <execution> >> >> <phase>package</phase> >> >> <goals> >> >> <goal>jarjar</goal> >> >> </goals> >> >> <configuration> >> >> <includes> >> >> <include>*org.apache.thrift:libthrift*</include> >> >> </includes> >> >> <rules> >> >> <rule> >> >> <pattern>org.apache.thrift.**</pattern> >> >> <result>com.xxx.JDBC.class.pattern.internal.thrift.@1 >> </result> >> >> </rule> >> >> <keep> >> >> <pattern>com.xxx.JDBC.class.pattern.**</pattern> >> >> </keep> >> >> </rules> >> >> </configuration> >> >> </execution> >> >> </executions> >> >> </plugin> >> >> >> On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <thelabd...@gmail.com>wrote: >> >>> Hi Zhang, >>> >>> I have the same issue in that I use some Cassandra client API's that >>> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted >>> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what >>> I added to the build.xml file for the hcatalog-pig-adapter project: >>> >>> <target name="jarjar"> >>> <echo message="basedir=${basedir}"/> >>> <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar" >>> todir="${basedir}/deps"/> >>> <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar" >>> todir="${basedir}/deps"/> >>> <copy todir="${basedir}/deps"> >>> <fileset dir="${basedir}/../../build/dist/lib" >>> includes="hive-*.jar"/> >>> </copy> >>> <taskdef name="jarjar" >>> classname="com.tonicsystems.jarjar.JarJarTask" >>> classpath="lib/jarjar-1.4.jar"/> >>> <jarjar jarfile="hcat-pig-shaded.jar"> >>> <zipgroupfileset dir="deps" includes="*.jar"/> >>> <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1 >>> "/> >>> </jarjar> >>> </target> >>> >>> I basically cooked up a deps directory that contains all the JARs that >>> the hcatalog-pig-adapter depends on and then used jarjar to create a single >>> jar with the thrift classes renamed at the bytecode level to >>> org.shaded.thrift >>> >>> You can do something similar for the JDBC code. >>> >>> Cheers, >>> Tim >>> >>> >>> >>> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo >>> <edlinuxg...@gmail.com>wrote: >>> >>>> You are kinda screwed. Thrift is wire-compatible in many cases but not >>>> API compatible. You can not have two applications build off two versions of >>>> thrift in the same classpath without something like OSGI. To insulate the >>>> class loaders from each other. >>>> >>>> Your best bet is upgrading "other component" to thrift 9 >>>> >>>> >>>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zhangxiaoyu...@gmail.com >>>> > wrote: >>>> >>>>> Hi, all, >>>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The >>>>> JDBC code depends on libthrift 0.9. However one component which depends on >>>>> libthrift 0.7 and not binary compatible with libthrift 0.9. >>>>> >>>>> When I downgrade to 0.7, I got below NoClassDefFoundError: >>>>> org/apache/thrift/scheme/StandardScheme. >>>>> >>>>> Any idea how to make HS2 working with thrift 0.7? Thanks. >>>>> >>>>> Johnny >>>>> >>>> >>>> >>> >> >