Thanks, Ted. I'll follow up with the Hive folks. Cheers, -Rick
Ted Yu <yuzhih...@gmail.com> wrote on 09/22/2015 03:41:12 PM: > From: Ted Yu <yuzhih...@gmail.com> > To: Richard Hillegas/San Francisco/IBM@IBMUS > Cc: Dev <dev@spark.apache.org> > Date: 09/22/2015 03:41 PM > Subject: Re: Derby version in Spark > > I cloned Hive 1.2 code base and saw: > > <derby.version>10.10.2.0</derby.version> > > So the version used by Spark is quite close to what Hive uses. > > On Tue, Sep 22, 2015 at 3:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: > I see. > I use maven to build so I observe different contents under > lib_managed directory. > > Here is snippet of dependency tree: > > [INFO] | +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark:compile > [INFO] | | +- com.jolbox:bonecp:jar:0.8.0.RELEASE:compile > [INFO] | | +- org.apache.derby:derby:jar:10.10.1.1:compile > > On Tue, Sep 22, 2015 at 3:21 PM, Richard Hillegas <rhil...@us.ibm.com> wrote: > Thanks, Ted. I'm working on my master branch. The lib_managed/jars > directory has a lot of jarballs, including hadoop and hive. Maybe > these were faulted in when I built with the following command? > > sbt/sbt -Phive assembly/assembly > > The Derby jars seem to be used in order to manage the metastore_db > database. Maybe my question should be directed to the Hive community? > > Thanks, > -Rick > > Here are the gory details: > > bash-3.2$ ls lib_managed/jars > FastInfoset-1.2.12.jar curator-test-2.4.0.jar jersey-test-framework- > grizzly2-1.9.jar parquet-format-2.3.0-incubating.jar > JavaEWAH-0.3.2.jar datanucleus-api-jdo-3.2.6.jar jets3t-0.7.1.jar > parquet-generator-1.7.0.jar > ST4-4.0.4.jar datanucleus-core-3.2.10.jar jetty-continuation-8.1. > 14.v20131031.jar parquet-hadoop-1.7.0.jar > activation-1.1.jar datanucleus-rdbms-3.2.9.jar jetty-http-8.1. > 14.v20131031.jar parquet-hadoop-bundle-1.6.0.jar > akka-actor_2.10-2.3.11.jar derby-10.10.1.1.jar jetty-io-8.1. > 14.v20131031.jar parquet-jackson-1.7.0.jar > akka-remote_2.10-2.3.11.jar derby-10.10.2.0.jar jetty-jndi-8.1. > 14.v20131031.jar platform-3.4.0.jar > akka-slf4j_2.10-2.3.11.jar genjavadoc-plugin_2.10.4-0.9-spark0.jar > jetty-plus-8.1.14.v20131031.jar pmml-agent-1.1.15.jar > akka-testkit_2.10-2.3.11.jar groovy-all-2.1.6.jar jetty-security-8. > 1.14.v20131031.jar pmml-model-1.1.15.jar > antlr-2.7.7.jar guava-11.0.2.jar jetty-server-8.1.14.v20131031.jar > pmml-schema-1.1.15.jar > antlr-runtime-3.4.jar guice-3.0.jar jetty-servlet-8.1. > 14.v20131031.jar postgresql-9.3-1102-jdbc41.jar > aopalliance-1.0.jar h2-1.4.183.jar jetty-util-6.1.26.jar py4j-0.8.2.1.jar > arpack_combined_all-0.1-javadoc.jar hadoop-annotations-2.2.0.jar > jetty-util-8.1.14.v20131031.jar pyrolite-4.4.jar > arpack_combined_all-0.1.jar hadoop-auth-2.2.0.jar jetty-webapp-8.1. > 14.v20131031.jar quasiquotes_2.10-2.0.0.jar > asm-3.2.jar hadoop-client-2.2.0.jar jetty-websocket-8.1. > 14.v20131031.jar reflectasm-1.07-shaded.jar > avro-1.7.4.jar hadoop-common-2.2.0.jar jetty-xml-8.1. > 14.v20131031.jar sac-1.3.jar > avro-1.7.7.jar hadoop-hdfs-2.2.0.jar jline-0.9.94.jar scala- > compiler-2.10.0.jar > avro-ipc-1.7.7-tests.jar hadoop-mapreduce-client-app-2.2.0.jar > jline-2.10.4.jar scala-compiler-2.10.4.jar > avro-ipc-1.7.7.jar hadoop-mapreduce-client-common-2.2.0.jar jline-2. > 12.jar scala-library-2.10.4.jar > avro-mapred-1.7.7-hadoop2.jar hadoop-mapreduce-client-core-2.2.0.jar > jna-3.4.0.jar scala-reflect-2.10.4.jar > breeze-macros_2.10-0.11.2.jar hadoop-mapreduce-client-jobclient-2.2. > 0.jar joda-time-2.5.jar scalacheck_2.10-1.11.3.jar > breeze_2.10-0.11.2.jar hadoop-mapreduce-client-shuffle-2.2.0.jar > jodd-core-3.5.2.jar scalap-2.10.0.jar > calcite-avatica-1.2.0-incubating.jar hadoop-yarn-api-2.2.0.jar > json-20080701.jar selenium-api-2.42.2.jar > calcite-core-1.2.0-incubating.jar hadoop-yarn-client-2.2.0.jar > json-20090211.jar selenium-chrome-driver-2.42.2.jar > calcite-linq4j-1.2.0-incubating.jar hadoop-yarn-common-2.2.0.jar > json4s-ast_2.10-3.2.10.jar selenium-firefox-driver-2.42.2.jar > cglib-2.2.1-v20090111.jar hadoop-yarn-server-common-2.2.0.jar > json4s-core_2.10-3.2.10.jar selenium-htmlunit-driver-2.42.2.jar > cglib-nodep-2.1_3.jar hadoop-yarn-server-nodemanager-2.2.0.jar > json4s-jackson_2.10-3.2.10.jar selenium-ie-driver-2.42.2.jar > chill-java-0.5.0.jar hamcrest-core-1.1.jar jsr173_api-1.0.jar > selenium-java-2.42.2.jar > chill_2.10-0.5.0.jar hamcrest-core-1.3.jar jsr305-1.3.9.jar > selenium-remote-driver-2.42.2.jar > commons-beanutils-1.7.0.jar hamcrest-library-1.3.jar jsr305-2.0. > 1.jar selenium-safari-driver-2.42.2.jar > commons-beanutils-core-1.8.0.jar hive-exec-1.2.1.spark.jar jta-1. > 1.jar selenium-support-2.42.2.jar > commons-cli-1.2.jar hive-metastore-1.2.1.spark.jar jtransforms-2.4. > 0.jar serializer-2.7.1.jar > commons-codec-1.10.jar htmlunit-2.14.jar jul-to-slf4j-1.7.10.jar > slf4j-api-1.7.10.jar > commons-codec-1.4.jar htmlunit-core-js-2.14.jar junit-4.10.jar > slf4j-log4j12-1.7.10.jar > commons-codec-1.5.jar httpclient-4.3.2.jar junit-dep-4.10.jar snappy-0.2.jar > commons-codec-1.9.jar httpcore-4.3.1.jar junit-dep-4.8.2.jar spire- > macros_2.10-0.7.4.jar > commons-collections-3.2.1.jar httpmime-4.3.2.jar junit-interface-0. > 10.jar spire_2.10-0.7.4.jar > commons-compiler-2.7.8.jar istack-commons-runtime-2.16.jar junit- > interface-0.9.jar stax-api-1.0.1.jar > commons-compress-1.4.1.jar ivy-2.4.0.jar libfb303-0.9.2.jar stream-2.7.0.jar > commons-configuration-1.6.jar jackson-core-asl-1.8.8.jar > libthrift-0.9.2.jar stringtemplate-3.2.1.jar > commons-dbcp-1.4.jar jackson-core-asl-1.9.13.jar lz4-1.3.0.jar > tachyon-client-0.7.1.jar > commons-digester-1.8.jar jackson-jaxrs-1.8.8.jar mesos-0.21.1- > shaded-protobuf.jar tachyon-underfs-hdfs-0.7.1.jar > commons-exec-1.1.jar jackson-mapper-asl-1.9.13.jar minlog-1.2.jar > tachyon-underfs-local-0.7.1.jar > commons-httpclient-3.1.jar jackson-xc-1.8.8.jar mockito-core-1.9. > 5.jar test-interface-0.5.jar > commons-io-2.1.jar janino-2.7.8.jar mysql-connector-java-5.1.34.jar > test-interface-1.0.jar > commons-io-2.4.jar jansi-1.4.jar nekohtml-1.9.20.jar uncommons- > maths-1.2.2a.jar > commons-lang-2.5.jar javassist-3.15.0-GA.jar netty-all-4.0. > 29.Final.jar unused-1.0.0.jar > commons-lang-2.6.jar javax.inject-1.jar objenesis-1.0.jar webbit-0.4.14.jar > commons-lang3-3.3.2.jar jaxb-api-2.2.2.jar objenesis-1.2.jar xalan-2.7.1.jar > commons-logging-1.1.3.jar jaxb-api-2.2.7.jar opencsv-2.3.jar > xercesImpl-2.11.0.jar > commons-math-2.1.jar jaxb-core-2.2.7.jar oro-2.0.8.jar xml-apis-1.4.01.jar > commons-math-2.2.jar jaxb-impl-2.2.3-1.jar paranamer-2.3.jar xmlenc-0.52.jar > commons-math3-3.4.1.jar jaxb-impl-2.2.7.jar paranamer-2.6.jar xz-1.0.jar > commons-net-3.1.jar jblas-1.2.4.jar parquet-avro-1.7.0.jar zookeeper-3.4.5.jar > commons-pool-1.5.4.jar jcl-over-slf4j-1.7.10.jar parquet-column-1.7.0.jar > core-1.1.2.jar jdo-api-3.0.1.jar parquet-common-1.7.0.jar > cssparser-0.9.13.jar jersey-guice-1.9.jar parquet-encoding-1.7.0.jar > > Ted Yu <yuzhih...@gmail.com> wrote on 09/22/2015 01:32:39 PM: > > > From: Ted Yu <yuzhih...@gmail.com> > > To: Richard Hillegas/San Francisco/IBM@IBMUS > > Cc: Dev <dev@spark.apache.org> > > Date: 09/22/2015 01:33 PM > > Subject: Re: Derby version in Spark > > > > > Which Spark release are you building ? > > > > For master branch, I get the following: > > > > lib_managed/jars/datanucleus-api-jdo-3.2.6.jar lib_managed/jars/ > > datanucleus-core-3.2.10.jar lib_managed/jars/datanucleus-rdbms-3.2.9.jar > > > > FYI > > > > On Tue, Sep 22, 2015 at 1:28 PM, Richard Hillegas <rhil...@us.ibm.com > > wrote: > > I see that lib_managed/jars holds these old Derby versions: > > > > lib_managed/jars/derby-10.10.1.1.jar > > lib_managed/jars/derby-10.10.2.0.jar > > > > The Derby 10.10 release family supports some ancient JVMs: Java SE 5 > > and Java ME CDC/Foundation Profile 1.1. It's hard to imagine anyone > > running Spark on the resource-constrained Java ME platform. Is Spark > > really deployed on Java SE 5? Is there some other reason that Spark > > uses the 10.10 Derby family? > > > > If no-one needs those ancient JVMs, maybe we could consider changing > > the Derby version to 10.11.1.1 or even to the upcoming 10.12.1.1 > > release (both run on Java 6 and up). > > > > Thanks, > > -Rick
