Yi Zhou created SPARK-15396: ------------------------------- Summary: [Spark] [SQL] It can't connect hive metastore database Key: SPARK-15396 URL: https://issues.apache.org/jira/browse/SPARK-15396 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.0 Reporter: Yi Zhou Priority: Critical
I am try to run Spark SQL using bin/spark-sql with Spark 2.0 master code(commit ba181c0c7a32b0e81bbcdbe5eed94fc97b58c83e) but ran across an issue that it always connect local derby database and can't connect my existing hive metastore database. Could you help me to check what's the root cause ? What's specific configuration for integration with hive metastore in Spark 2.0 ? BTW, this case is OK in Spark 1.6. Thanks in advance ! Build package command: ./dev/make-distribution.sh --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.5.1 -Phive -Phive-thriftserver -DskipTests Key configurations in spark-defaults.conf: spark.sql.hive.metastore.version=1.1.0 spark.sql.hive.metastore.jars=/usr/lib/hive/lib/*:/usr/lib/hadoop/client/* spark.executor.extraClassPath=/etc/hive/conf spark.driver.extraClassPath=/etc/hive/conf spark.yarn.jars=local:/usr/lib/spark/jars/* There is existing hive metastore database named by "test_sparksql". I always got error "metastore.ObjectStore: Failed to get database test_sparksql, returning NoSuchObjectException" after issuing 'use test_sparksql'. Please see below steps for details. $ /usr/lib/spark/bin/spark-sql --master yarn --deploy-mode client SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/avro/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/05/12 22:23:28 WARN conf.HiveConf: HiveConf of name hive.enable.spark.execution.engine does not exist 16/05/12 22:23:30 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 16/05/12 22:23:30 INFO metastore.ObjectStore: ObjectStore, initialize called 16/05/12 22:23:30 WARN DataNucleus.General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/hive/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/jars/datanucleus-rdbms-3.2.9.jar." 16/05/12 22:23:30 WARN DataNucleus.General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/hive/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/jars/datanucleus-core-3.2.10.jar." 16/05/12 22:23:30 WARN DataNucleus.General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark/jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/hive/lib/datanucleus-api-jdo-3.2.6.jar." 16/05/12 22:23:30 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 16/05/12 22:23:30 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 16/05/12 22:23:31 WARN conf.HiveConf: HiveConf of name hive.enable.spark.execution.engine does not exist 16/05/12 22:23:31 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 16/05/12 22:23:32 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:32 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:33 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:33 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:33 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 16/05/12 22:23:33 INFO metastore.ObjectStore: Initialized ObjectStore 16/05/12 22:23:33 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 16/05/12 22:23:33 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException 16/05/12 22:23:34 INFO metastore.HiveMetaStore: Added admin role in metastore 16/05/12 22:23:34 INFO metastore.HiveMetaStore: Added public role in metastore 16/05/12 22:23:34 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 16/05/12 22:23:34 INFO metastore.HiveMetaStore: 0: get_all_databases 16/05/12 22:23:34 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases 16/05/12 22:23:34 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=* 16/05/12 22:23:34 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=* 16/05/12 22:23:34 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:34 INFO session.SessionState: Created local directory: /tmp/4e7ccc40-e10b-455c-b51d-ed225be85ffe_resources 16/05/12 22:23:34 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4e7ccc40-e10b-455c-b51d-ed225be85ffe 16/05/12 22:23:34 INFO session.SessionState: Created local directory: /tmp/root/4e7ccc40-e10b-455c-b51d-ed225be85ffe 16/05/12 22:23:34 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4e7ccc40-e10b-455c-b51d-ed225be85ffe/_tmp_space.db 16/05/12 22:23:34 INFO spark.SparkContext: Running Spark version 2.0.0-SNAPSHOT 16/05/12 22:23:34 INFO spark.SecurityManager: Changing view acls to: root 16/05/12 22:23:34 INFO spark.SecurityManager: Changing modify acls to: root 16/05/12 22:23:34 INFO spark.SecurityManager: Changing view acls groups to: 16/05/12 22:23:34 INFO spark.SecurityManager: Changing modify acls groups to: 16/05/12 22:23:34 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 16/05/12 22:23:35 INFO util.Utils: Successfully started service 'sparkDriver' on port 37223. 16/05/12 22:23:35 INFO spark.SparkEnv: Registering MapOutputTracker 16/05/12 22:23:35 INFO spark.SparkEnv: Registering BlockManagerMaster 16/05/12 22:23:35 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-5a30adbe-4f9a-4b34-b52f-b61671f8b06d 16/05/12 22:23:35 INFO memory.MemoryStore: MemoryStore started with capacity 511.1 MB 16/05/12 22:23:35 INFO spark.SparkEnv: Registering OutputCommitCoordinator 16/05/12 22:23:35 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/05/12 22:23:35 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 16/05/12 22:23:35 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 16/05/12 22:23:35 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.3.11:4040 16/05/12 22:23:35 INFO client.RMProxy: Connecting to ResourceManager at hw-node2/192.168.3.12:8032 16/05/12 22:23:35 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers 16/05/12 22:23:35 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (196608 MB per container) 16/05/12 22:23:35 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 16/05/12 22:23:35 INFO yarn.Client: Setting up container launch context for our AM 16/05/12 22:23:35 INFO yarn.Client: Setting up the launch environment for our AM container 16/05/12 22:23:35 INFO yarn.Client: Preparing resources for our AM container 16/05/12 22:23:35 INFO yarn.Client: Uploading resource file:/tmp/spark-a712ffb6-a0d0-48db-99b4-ee6a41b3f132/__spark_conf__7597761027449817951.zip -> hdfs://hw-node2:8020/user/root/.sparkStaging/application_1463053929123_0006/__spark_conf__.zip 16/05/12 22:23:36 INFO yarn.Client: Uploading resource file:/tmp/spark-a712ffb6-a0d0-48db-99b4-ee6a41b3f132/__spark_conf__9093112552235548615.zip -> hdfs://hw-node2:8020/user/root/.sparkStaging/application_1463053929123_0006/__spark_conf__9093112552235548615.zip 16/05/12 22:23:36 INFO spark.SecurityManager: Changing view acls to: root 16/05/12 22:23:36 INFO spark.SecurityManager: Changing modify acls to: root 16/05/12 22:23:36 INFO spark.SecurityManager: Changing view acls groups to: 16/05/12 22:23:36 INFO spark.SecurityManager: Changing modify acls groups to: 16/05/12 22:23:36 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 16/05/12 22:23:36 INFO yarn.Client: Submitting application application_1463053929123_0006 to ResourceManager 16/05/12 22:23:36 INFO impl.YarnClientImpl: Submitted application application_1463053929123_0006 16/05/12 22:23:36 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1463053929123_0006 and attemptId None 16/05/12 22:23:37 INFO yarn.Client: Application report for application_1463053929123_0006 (state: ACCEPTED) 16/05/12 22:23:37 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.root start time: 1463063016173 final status: UNDEFINED tracking URL: http://hw-node2:8088/proxy/application_1463053929123_0006/ user: root 16/05/12 22:23:38 INFO yarn.Client: Application report for application_1463053929123_0006 (state: ACCEPTED) 16/05/12 22:23:38 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null) 16/05/12 22:23:38 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hw-node2, PROXY_URI_BASES -> http://hw-node2:8088/proxy/application_1463053929123_0006), /proxy/application_1463053929123_0006 16/05/12 22:23:38 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 16/05/12 22:23:39 INFO yarn.Client: Application report for application_1463053929123_0006 (state: RUNNING) 16/05/12 22:23:39 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.3.16 ApplicationMaster RPC port: 0 queue: root.root start time: 1463063016173 final status: UNDEFINED tracking URL: http://hw-node2:8088/proxy/application_1463053929123_0006/ user: root 16/05/12 22:23:39 INFO cluster.YarnClientSchedulerBackend: Application application_1463053929123_0006 has started running. 16/05/12 22:23:39 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45022. 16/05/12 22:23:39 INFO netty.NettyBlockTransferService: Server created on 192.168.3.11:45022 16/05/12 22:23:39 INFO storage.BlockManager: external shuffle service port = 7337 16/05/12 22:23:39 INFO storage.BlockManagerMaster: Trying to register BlockManager 16/05/12 22:23:39 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.3.11:45022 with 511.1 MB RAM, BlockManagerId(driver, 192.168.3.11, 45022) 16/05/12 22:23:39 INFO storage.BlockManagerMaster: Registered BlockManager 16/05/12 22:23:39 INFO scheduler.EventLoggingListener: Logging events to hdfs://hw-node2:8020/user/spark/applicationHistory/application_1463053929123_0006 16/05/12 22:23:39 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 16/05/12 22:23:39 INFO hive.HiveSharedState: Setting Hive metastore warehouse path to '/root/spark-warehouse' 16/05/12 22:23:39 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.1.0 using file:/usr/lib/hive/lib/httpcore-4.2.5.jar:file:/usr/lib/hive/lib/hive-contrib.jar:file:/usr/lib/hive/lib/oro-2.0.8.jar:file:/usr/lib/hive/lib/accumulo-start-1.6.0.jar:file:/usr/lib/hive/lib/groovy-all-2.4.4.jar:file:/usr/lib/hive/lib/hive-metastore.jar:file:/usr/lib/hive/lib/hive-beeline.jar:file:/usr/lib/hive/lib/datanucleus-core-3.2.10.jar:file:/usr/lib/hive/lib/jackson-core-2.2.2.jar:file:/usr/lib/hive/lib/velocity-1.5.jar:file:/usr/lib/hive/lib/hive-serde.jar:file:/usr/lib/hive/lib/hive-metastore-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/commons-beanutils-core-1.8.0.jar:file:/usr/lib/hive/lib/hamcrest-core-1.1.jar:file:/usr/lib/hive/lib/jta-1.1.jar:file:/usr/lib/hive/lib/hive-shims-0.23-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/antlr-2.7.7.jar:file:/usr/lib/hive/lib/hive-exec-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:file:/usr/lib/hive/lib/accumulo-fate-1.6.0.jar:file:/usr/lib/hive/lib/hive-accumulo-handler.jar:file:/usr/lib/hive/lib/snappy-java-1.0.4.1.jar:file:/usr/lib/hive/lib/tempus-fugit-1.1.jar:file:/usr/lib/hive/lib/maven-scm-provider-svn-commons-1.4.jar:file:/usr/lib/hive/lib/libfb303-0.9.2.jar:file:/usr/lib/hive/lib/datanucleus-rdbms-3.2.9.jar:file:/usr/lib/hive/lib/xz-1.0.jar:file:/usr/lib/hive/lib/hbase-common.jar:file:/usr/lib/hive/lib/activation-1.1.jar:file:/usr/lib/hive/lib/hive-ant-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/accumulo-trace-1.6.0.jar:file:/usr/lib/hive/lib/hive-serde-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/commons-compress-1.4.1.jar:file:/usr/lib/hive/lib/hbase-hadoop2-compat.jar:file:/usr/lib/hive/lib/commons-configuration-1.6.jar:file:/usr/lib/hive/lib/servlet-api-2.5.jar:file:/usr/lib/hive/lib/libthrift-0.9.2.jar:file:/usr/lib/hive/lib/stax-api-1.0.1.jar:file:/usr/lib/hive/lib/hive-testutils.jar:file:/usr/lib/hive/lib/hive-shims-scheduler-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hive-testutils-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/junit-4.11.jar:file:/usr/lib/hive/lib/jackson-annotations-2.2.2.jar:file:/usr/lib/hive/lib/stringtemplate-3.2.1.jar:file:/usr/lib/hive/lib/super-csv-2.2.0.jar:file:/usr/lib/hive/lib/hive-hwi-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/log4j-1.2.16.jar:file:/usr/lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:file:/usr/lib/hive/lib/accumulo-core-1.6.0.jar:file:/usr/lib/hive/lib/hive-hbase-handler.jar:file:/usr/lib/hive/lib/high-scale-lib-1.1.1.jar:file:/usr/lib/hive/lib/hbase-protocol.jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hive-jdbc.jar:file:/usr/lib/hive/lib/commons-logging-1.1.3.jar:file:/usr/lib/hive/lib/derby-10.11.1.1.jar:file:/usr/lib/hive/lib/hive-jdbc-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hive-shims-scheduler.jar:file:/usr/lib/hive/lib/asm-commons-3.1.jar:file:/usr/lib/hive/lib/hive-jdbc-standalone.jar:file:/usr/lib/hive/lib/maven-scm-api-1.4.jar:file:/usr/lib/hive/lib/janino-2.7.6.jar:file:/usr/lib/hive/lib/hive-cli.jar:file:/usr/lib/hive/lib/maven-scm-provider-svnexe-1.4.jar:file:/usr/lib/hive/lib/bonecp-0.8.0.RELEASE.jar:file:/usr/lib/hive/lib/zookeeper.jar:file:/usr/lib/hive/lib/jline-2.12.jar:file:/usr/lib/hive/lib/asm-3.2.jar:file:/usr/lib/hive/lib/logredactor-1.0.3.jar:file:/usr/lib/hive/lib/hive-ant.jar:file:/usr/lib/hive/lib/hive-shims-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/ant-launcher-1.9.1.jar:file:/usr/lib/hive/lib/hive-cli-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/gson-2.2.4.jar:file:/usr/lib/hive/lib/avro.jar:file:/usr/lib/hive/lib/parquet-hadoop-bundle.jar:file:/usr/lib/hive/lib/commons-beanutils-1.7.0.jar:file:/usr/lib/hive/lib/commons-digester-1.8.jar:file:/usr/lib/hive/lib/apache-log4j-extras-1.2.17.jar:file:/usr/lib/hive/lib/calcite-core-1.0.0-incubating.jar:file:/usr/lib/hive/lib/metrics-json-3.0.2.jar:file:/usr/lib/hive/lib/hive-jdbc-1.1.0-cdh5.5.1-standalone.jar:file:/usr/lib/hive/lib/jackson-databind-2.2.2.jar:file:/usr/lib/hive/lib/hive-exec.jar:file:/usr/lib/hive/lib/jersey-server-1.14.jar:file:/usr/lib/hive/lib/asm-tree-3.1.jar:file:/usr/lib/hive/lib/jdo-api-3.0.1.jar:file:/usr/lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:file:/usr/lib/hive/lib/metrics-core-3.0.2.jar:file:/usr/lib/hive/lib/commons-dbcp-1.4.jar:file:/usr/lib/hive/lib/mail-1.4.1.jar:file:/usr/lib/hive/lib/metrics-jvm-3.0.2.jar:file:/usr/lib/hive/lib/paranamer-2.3.jar:file:/usr/lib/hive/lib/commons-lang-2.6.jar:file:/usr/lib/hive/lib/commons-compiler-2.7.6.jar:file:/usr/lib/hive/lib/commons-codec-1.4.jar:file:/usr/lib/hive/lib/guava-14.0.1.jar:file:/usr/lib/hive/lib/hive-service-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/jersey-servlet-1.14.jar:file:/usr/lib/hive/lib/regexp-1.3.jar:file:/usr/lib/hive/lib/jpam-1.1.jar:file:/usr/lib/hive/lib/calcite-linq4j-1.0.0-incubating.jar:file:/usr/lib/hive/lib/hive-accumulo-handler-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hbase-server.jar:file:/usr/lib/hive/lib/eigenbase-properties-1.1.4.jar:file:/usr/lib/hive/lib/commons-pool-1.5.4.jar:file:/usr/lib/hive/lib/commons-vfs2-2.0.jar:file:/usr/lib/hive/lib/jackson-jaxrs-1.9.2.jar:file:/usr/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/commons-math-2.1.jar:file:/usr/lib/hive/lib/commons-cli-1.2.jar:file:/usr/lib/hive/lib/commons-io-2.4.jar:file:/usr/lib/hive/lib/ant-1.9.1.jar:file:/usr/lib/hive/lib/ST4-4.0.4.jar:file:/usr/lib/hive/lib/hive-shims-common-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hive-common.jar:file:/usr/lib/hive/lib/jetty-all-server-7.6.0.v20120127.jar:file:/usr/lib/hive/lib/hive-service.jar:file:/usr/lib/hive/lib/hbase-hadoop-compat.jar:file:/usr/lib/hive/lib/hive-shims-0.23.jar:file:/usr/lib/hive/lib/hive-contrib-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/curator-client-2.6.0.jar:file:/usr/lib/hive/lib/commons-httpclient-3.0.1.jar:file:/usr/lib/hive/lib/plexus-utils-1.5.6.jar:file:/usr/lib/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:file:/usr/lib/hive/lib/jetty-all-7.6.0.v20120127.jar:file:/usr/lib/hive/lib/hive-shims.jar:file:/usr/lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:file:/usr/lib/hive/lib/htrace-core.jar:file:/usr/lib/hive/lib/httpclient-4.2.5.jar:file:/usr/lib/hive/lib/jcommander-1.32.jar:file:/usr/lib/hive/lib/antlr-runtime-3.4.jar:file:/usr/lib/hive/lib/opencsv-2.3.jar:file:/usr/lib/hive/lib/jsr305-3.0.0.jar:file:/usr/lib/hive/lib/jackson-xc-1.9.2.jar:file:/usr/lib/hive/lib/hive-shims-common.jar:file:/usr/lib/hive/lib/curator-framework-2.6.0.jar:file:/usr/lib/hive/lib/calcite-avatica-1.0.0-incubating.jar:file:/usr/lib/hive/lib/hive-beeline-1.1.0-cdh5.5.1.jar:file:/usr/lib/hive/lib/hive-hwi.jar:file:/usr/lib/hive/lib/hbase-client.jar:file:/usr/lib/hadoop/client/httpcore-4.2.5.jar:file:/usr/lib/hadoop/client/hadoop-hdfs.jar:file:/usr/lib/hadoop/client/apacheds-i18n-2.0.0-M15.jar:file:/usr/lib/hadoop/client/apacheds-kerberos-codec.jar:file:/usr/lib/hadoop/client/slf4j-api-1.7.5.jar:file:/usr/lib/hadoop/client/commons-net.jar:file:/usr/lib/hadoop/client/commons-beanutils-core-1.8.0.jar:file:/usr/lib/hadoop/client/jackson-annotations-2.2.3.jar:file:/usr/lib/hadoop/client/commons-logging.jar:file:/usr/lib/hadoop/client/curator-recipes-2.7.1.jar:file:/usr/lib/hadoop/client/hadoop-aws-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/snappy-java.jar:file:/usr/lib/hadoop/client/leveldbjni-all.jar:file:/usr/lib/hadoop/client/jackson-databind.jar:file:/usr/lib/hadoop/client/commons-lang.jar:file:/usr/lib/hadoop/client/xmlenc-0.52.jar:file:/usr/lib/hadoop/client/snappy-java-1.0.4.1.jar:file:/usr/lib/hadoop/client/commons-httpclient.jar:file:/usr/lib/hadoop/client/hadoop-yarn-server-common.jar:file:/usr/lib/hadoop/client/jackson-databind-2.2.3.jar:file:/usr/lib/hadoop/client/guava-11.0.2.jar:file:/usr/lib/hadoop/client/xz-1.0.jar:file:/usr/lib/hadoop/client/hadoop-yarn-api-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-jobclient.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-app.jar:file:/usr/lib/hadoop/client/activation-1.1.jar:file:/usr/lib/hadoop/client/jaxb-api.jar:file:/usr/lib/hadoop/client/commons-compress-1.4.1.jar:file:/usr/lib/hadoop/client/commons-configuration-1.6.jar:file:/usr/lib/hadoop/client/jackson-xc.jar:file:/usr/lib/hadoop/client/servlet-api-2.5.jar:file:/usr/lib/hadoop/client/xmlenc.jar:file:/usr/lib/hadoop/client/jackson-jaxrs.jar:file:/usr/lib/hadoop/client/jackson-xc-1.8.8.jar:file:/usr/lib/hadoop/client/hadoop-common-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/apacheds-kerberos-codec-2.0.0-M15.jar:file:/usr/lib/hadoop/client/commons-cli.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-app-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/aws-java-sdk-1.7.4.jar:file:/usr/lib/hadoop/client/netty.jar:file:/usr/lib/hadoop/client/protobuf-java.jar:file:/usr/lib/hadoop/client/jaxb-api-2.2.2.jar:file:/usr/lib/hadoop/client/commons-logging-1.1.3.jar:file:/usr/lib/hadoop/client/commons-net-3.1.jar:file:/usr/lib/hadoop/client/hadoop-annotations.jar:file:/usr/lib/hadoop/client/hadoop-hdfs-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/jersey-core.jar:file:/usr/lib/hadoop/client/hadoop-yarn-client-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/hadoop-auth-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/zookeeper.jar:file:/usr/lib/hadoop/client/commons-collections-3.2.2.jar:file:/usr/lib/hadoop/client/servlet-api.jar:file:/usr/lib/hadoop/client/guava.jar:file:/usr/lib/hadoop/client/hadoop-yarn-api.jar:file:/usr/lib/hadoop/client/commons-math3.jar:file:/usr/lib/hadoop/client/slf4j-api.jar:file:/usr/lib/hadoop/client/stax-api.jar:file:/usr/lib/hadoop/client/hadoop-auth.jar:file:/usr/lib/hadoop/client/commons-io.jar:file:/usr/lib/hadoop/client/commons-digester.jar:file:/usr/lib/hadoop/client/hadoop-annotations-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/gson-2.2.4.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-core-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/avro.jar:file:/usr/lib/hadoop/client/activation.jar:file:/usr/lib/hadoop/client/apacheds-i18n.jar:file:/usr/lib/hadoop/client/hadoop-yarn-common.jar:file:/usr/lib/hadoop/client/commons-beanutils-1.7.0.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-common.jar:file:/usr/lib/hadoop/client/commons-digester-1.8.jar:file:/usr/lib/hadoop/client/jetty-util.jar:file:/usr/lib/hadoop/client/jackson-core-asl-1.8.8.jar:file:/usr/lib/hadoop/client/jetty-util-6.1.26.cloudera.4.jar:file:/usr/lib/hadoop/client/httpcore.jar:file:/usr/lib/hadoop/client/curator-client.jar:file:/usr/lib/hadoop/client/netty-3.6.2.Final.jar:file:/usr/lib/hadoop/client/jackson-mapper-asl.jar:file:/usr/lib/hadoop/client/commons-beanutils-core.jar:file:/usr/lib/hadoop/client/jackson-jaxrs-1.8.8.jar:file:/usr/lib/hadoop/client/xz.jar:file:/usr/lib/hadoop/client/paranamer-2.3.jar:file:/usr/lib/hadoop/client/commons-lang-2.6.jar:file:/usr/lib/hadoop/client/jackson-annotations.jar:file:/usr/lib/hadoop/client/commons-codec-1.4.jar:file:/usr/lib/hadoop/client/jersey-core-1.9.jar:file:/usr/lib/hadoop/client/api-asn1-api-1.0.0-M20.jar:file:/usr/lib/hadoop/client/commons-collections.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/api-util.jar:file:/usr/lib/hadoop/client/jsr305.jar:file:/usr/lib/hadoop/client/httpclient.jar:file:/usr/lib/hadoop/client/xml-apis-1.3.04.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-core.jar:file:/usr/lib/hadoop/client/curator-client-2.7.1.jar:file:/usr/lib/hadoop/client/commons-httpclient-3.1.jar:file:/usr/lib/hadoop/client/hadoop-yarn-common-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/commons-cli-1.2.jar:file:/usr/lib/hadoop/client/commons-io-2.4.jar:file:/usr/lib/hadoop/client/curator-framework.jar:file:/usr/lib/hadoop/client/stax-api-1.0-2.jar:file:/usr/lib/hadoop/client/htrace-core4.jar:file:/usr/lib/hadoop/client/jackson-core-2.2.3.jar:file:/usr/lib/hadoop/client/jackson-core-asl.jar:file:/usr/lib/hadoop/client/commons-configuration.jar:file:/usr/lib/hadoop/client/commons-compress.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-common-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/xercesImpl-2.9.1.jar:file:/usr/lib/hadoop/client/jersey-client-1.9.jar:file:/usr/lib/hadoop/client/log4j.jar:file:/usr/lib/hadoop/client/jackson-mapper-asl-1.8.8.jar:file:/usr/lib/hadoop/client/leveldbjni-all-1.8.jar:file:/usr/lib/hadoop/client/api-util-1.0.0-M20.jar:file:/usr/lib/hadoop/client/curator-framework-2.7.1.jar:file:/usr/lib/hadoop/client/commons-codec.jar:file:/usr/lib/hadoop/client/xml-apis.jar:file:/usr/lib/hadoop/client/jersey-client.jar:file:/usr/lib/hadoop/client/hadoop-yarn-client.jar:file:/usr/lib/hadoop/client/aws-java-sdk.jar:file:/usr/lib/hadoop/client/paranamer.jar:file:/usr/lib/hadoop/client/hadoop-aws.jar:file:/usr/lib/hadoop/client/hadoop-yarn-server-common-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/commons-math3-3.1.1.jar:file:/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.5.1.jar:file:/usr/lib/hadoop/client/httpclient-4.2.5.jar:file:/usr/lib/hadoop/client/commons-beanutils.jar:file:/usr/lib/hadoop/client/hadoop-common.jar:file:/usr/lib/hadoop/client/api-asn1-api.jar:file:/usr/lib/hadoop/client/htrace-core4-4.0.1-incubating.jar:file:/usr/lib/hadoop/client/log4j-1.2.17.jar:file:/usr/lib/hadoop/client/jsr305-3.0.0.jar:file:/usr/lib/hadoop/client/curator-recipes.jar:file:/usr/lib/hadoop/client/slf4j-log4j12.jar:file:/usr/lib/hadoop/client/jackson-core.jar:file:/usr/lib/hadoop/client/protobuf-java-2.5.0.jar:file:/usr/lib/hadoop/client/xercesImpl.jar:file:/usr/lib/hadoop/client/gson.jar 16/05/12 22:23:40 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 16/05/12 22:23:40 INFO metastore.ObjectStore: ObjectStore, initialize called 16/05/12 22:23:40 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 16/05/12 22:23:40 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 16/05/12 22:23:41 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 16/05/12 22:23:42 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:42 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:42 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:42 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:42 INFO DataNucleus.Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 16/05/12 22:23:42 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 16/05/12 22:23:42 INFO metastore.ObjectStore: Initialized ObjectStore 16/05/12 22:23:42 INFO metastore.HiveMetaStore: Added admin role in metastore 16/05/12 22:23:42 INFO metastore.HiveMetaStore: Added public role in metastore 16/05/12 22:23:42 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 16/05/12 22:23:42 INFO log.PerfLogger: <PERFLOG method=get_all_functions from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> 16/05/12 22:23:42 INFO metastore.HiveMetaStore: 0: get_all_functions 16/05/12 22:23:42 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_functions 16/05/12 22:23:42 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 16/05/12 22:23:42 INFO log.PerfLogger: </PERFLOG method=get_all_functions start=1463063022896 end=1463063022941 duration=45 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0 retryCount=0 error=false> 16/05/12 22:23:42 INFO session.SessionState: Created local directory: /tmp/f1ff20d6-3eac-4df0-adbd-64f7e73f35e8_resources 16/05/12 22:23:42 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/f1ff20d6-3eac-4df0-adbd-64f7e73f35e8 16/05/12 22:23:42 INFO session.SessionState: Created local directory: /tmp/root/f1ff20d6-3eac-4df0-adbd-64f7e73f35e8 16/05/12 22:23:42 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/f1ff20d6-3eac-4df0-adbd-64f7e73f35e8/_tmp_space.db 16/05/12 22:23:42 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 16/05/12 22:23:42 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.1.0) is /root/spark-warehouse 16/05/12 22:23:43 INFO session.SessionState: Created local directory: /tmp/4f466b18-e85b-4fa5-9b3a-2a1a67118851_resources 16/05/12 22:23:43 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4f466b18-e85b-4fa5-9b3a-2a1a67118851 16/05/12 22:23:43 INFO session.SessionState: Created local directory: /tmp/root/4f466b18-e85b-4fa5-9b3a-2a1a67118851 16/05/12 22:23:43 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4f466b18-e85b-4fa5-9b3a-2a1a67118851/_tmp_space.db 16/05/12 22:23:43 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 16/05/12 22:23:43 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.1.0) is /root/spark-warehouse spark-sql> use test_sparksql; 16/05/12 22:25:06 INFO execution.SparkSqlParser: Parsing command: use test_sparksql 16/05/12 22:25:06 INFO log.PerfLogger: <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> 16/05/12 22:25:06 INFO metastore.HiveMetaStore: 0: create_database: Database(name:default, description:default database, locationUri:hdfs://hw-node2:8020/root/spark-warehouse, parameters:{}) 16/05/12 22:25:06 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:hdfs://hw-node2:8020/root/spark-warehouse, parameters:{}) 16/05/12 22:25:06 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:898) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:133) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) at com.sun.proxy.$Proxy34.create_database(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:645) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91) at com.sun.proxy.$Proxy35.createDatabase(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:341) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply$mcV$sp(HiveClientImpl.scala:292) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply(HiveClientImpl.scala:292) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply(HiveClientImpl.scala:292) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:263) at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210) at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209) at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:252) at org.apache.spark.sql.hive.client.HiveClientImpl.createDatabase(HiveClientImpl.scala:291) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply$mcV$sp(HiveExternalCatalog.scala:94) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply(HiveExternalCatalog.scala:94) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply(HiveExternalCatalog.scala:94) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:68) at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:93) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:142) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:84) at org.apache.spark.sql.hive.HiveSessionCatalog.<init>(HiveSessionCatalog.scala:50) at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:49) at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48) at org.apache.spark.sql.hive.HiveSessionState$$anon$1.<init>(HiveSessionState.scala:63) at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63) at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:48) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:61) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541) at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:671) at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:325) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:240) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/05/12 22:25:06 INFO log.PerfLogger: </PERFLOG method=create_database start=1463063106660 end=1463063106665 duration=5 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0 retryCount=-1 error=true> 16/05/12 22:25:06 INFO log.PerfLogger: <PERFLOG method=get_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> 16/05/12 22:25:06 INFO metastore.HiveMetaStore: 0: get_database: test_sparksql 16/05/12 22:25:06 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_database: test_sparksql 16/05/12 22:25:06 WARN metastore.ObjectStore: Failed to get database test_sparksql, returning NoSuchObjectException 16/05/12 22:25:06 INFO log.PerfLogger: </PERFLOG method=get_database start=1463063106947 end=1463063106950 duration=3 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0 retryCount=-1 error=true> Error in query: Database 'test_sparksql' not found; -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org