[ https://issues.apache.org/jira/browse/SPARK-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159692#comment-17159692 ]
xiepengjie edited comment on SPARK-15118 at 7/17/20, 5:52 AM: -------------------------------------------------------------- Maybe we should merge HiveConf too: org.apache.spark.sql.internal.BaseSessionStateBuilder#conf {code:java} protected lazy val conf: SQLConf ={ val conf = parentState.map(_.conf.clone()).getOrElse(new SQLConf) mergeSparkConf(conf, session.sparkContext.conf) // Spark Sql support properties from hive-site.xml mergeHiveConf(conf, new HiveConf) conf } {code} was (Author: xiepengjie): Maybe we should merge HiveConf too: org.apache.spark.sql.internal.BaseSessionStateBuilder#conf {code:java} protected lazy val conf: SQLConf ={ val conf = parentState.map(_.conf.clone()).getOrElse(new SQLConf) mergeSparkConf(conf, session.sparkContext.conf) // Spark Sql support properties from hive-site.xml mergeHiveConf(conf, new HiveConf); conf } {code} > spark couldn't get hive properyties in hive-site.xml > ----------------------------------------------------- > > Key: SPARK-15118 > URL: https://issues.apache.org/jira/browse/SPARK-15118 > Project: Spark > Issue Type: Bug > Components: Block Manager, Deploy, Spark Core > Affects Versions: 1.6.1 > Environment: hadoop-2.7.1.tar.gz; > apache-hive-2.0.0-bin.tar.gz; > spark-1.6.1-bin-hadoop2.6.tgz; > scala-2.11.8.tgz > Reporter: eksmile > Priority: Major > > I have three question. > First: > I've already put "hive-site.xml" in $SPARK_HOME/conf, but when I run > spark-sql, it tell me "HiveConf of name *** does not exist", and repeat many > times. > All of these "HiveConf" are in "hive-site.xml", why these warnings appear? > I'm not sure this is a bug or not. > Second: > In the middle of logs as follow, there's a paragraph : "Failed to get > database default, returning NoSuchObjectException", > I don't know is there something worng? > Third: > In the middle of logs, there's a paragraph : " metastore.MetaStoreDirectSql: > Using direct SQL, underlying DB is DERBY", > but, in the end of logs, there's a paragraph : "metastore.MetaStoreDirectSql: > Using direct SQL, underlying DB is MYSQL" > My Hive metastore is MYSQL. Is this something wrong? > spark-env.sh as follow: > export JAVA_HOME=/usr/java/jdk1.8.0_73 > export SCALA_HOME=/home/scala > export SPARK_MASTER_IP=192.168.124.129 > export SPARK_WORKER_MEMORY=1g > export HADOOP_CONF_DIR=/usr/hadoop/etc/hadoop > export HIVE_HOME=/opt/hive > export HIVE_CONF_DIR=/opt/hive/conf > export > SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/hive/lib/mysql-connector-java-5.1.38-bin.jar > export HADOOP_HOME=/usr/hadoop > Thanks for reading > Here're the logs: > [yezt@Master spark]$ bin/spark-sql --master spark://master:7077 > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.orc.time.counters does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.metrics.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.timeout.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.connect.retry.limit does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.direct does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.enforce.stats does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.client.consistent.splits does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.tez.session.lifetime does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.timedout.txn.reaper.start does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.cache.ttl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.management.acl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.delegation.token.lifetime does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.strict.checks.large.query does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.min does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.user does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.wait.queue.comparator.class.name does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.am.liveness.heartbeat.interval.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.object.cache.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.use.ssl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.metastore.local > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.service.metrics.file.location does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.retry.delay.seconds does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.num.file.cleaner.threads does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.test.fail.compaction does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.service.metrics.class does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.vectorized.execution.reducesink.new.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.max.num.delta does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.retention.attempted does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.port does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.initiator.failed.compacts.threshold does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.service.metrics.reporter does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.execution.mode does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.limittranspose does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.memory.mode does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.threadpool.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.use.spnego does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.service.metrics.file.frequency does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.fastpath does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.file.cleanup.delay.seconds does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.management.rpc.port does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.enforce.tree does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.retention.failed does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.host does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.orc.splits.ms.footer.cache.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.point.lookup.min does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.service.refresh.interval.sec does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.max.output.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.driver.parallel.compilation does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.tez.bucket.pruning does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.cache.allow.synthetic.fileid does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.hash.table.inflation.factor does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggr.stats.hbase.ttl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.enforce.vectorized does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.exec.schema.evolution does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.allow.uber does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.llap.concurrent.queries does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.llap.auto.auth > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.orc.splits.include.fileid does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.communicator.num.threads does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.communicator.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.max.partitions does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.yarn.shuffle.port does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.http.response.header.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.conf.internal.variable.list does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.limittranspose.reductionpercentage does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.llap.io.enabled > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.retry.limit does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.orc.splits.directory.batch.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.cache.max.reader.wait does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.reenable.max.timeout.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.execution.mode > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.catalog.cache.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.task.scheduler.wait.queue.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggr.stats.cache.entries does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggr.stats.memory.ttl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.rpc.port does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.cache.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.cache.clean.until does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.metastore.port > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.spark.dynamic.partition.pruning does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.metrics.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.cli.tez.session.async does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.tez.bucket.pruning.compat does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.spnego.principal does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.shuffle.dir.watcher.enabled does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.arena.count does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.communicator.connection.timeout.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.transpose.aggr.join does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.spark.dynamic.partition.pruning.max.data.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggr.stats.invalidator.frequency does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.llap.io.use.lrfu > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.conf.hidden.list > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.timedout.txn.reaper.interval does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.max.variance does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.lrfu.lambda does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.point.lookup does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.web.ssl does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.retention.succeeded does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.use.fileid.path does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.mapjoin.optimized.hashtable.probe.percent does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.reenable.min.timeout.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.support.special.characters.tablename does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.yarn.container.mb does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.dynamic.partition.hashjoin does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.task.scheduler.enable.preemption does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.num.executors does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.cache.max.full does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.connection.class does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.password does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.cache.max.writer.wait does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.http.request.header.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.webui.max.threads does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.limittranspose.reductiontuples does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.test.rollbacktxn > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.num.schedulable.tasks.per.node does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name hive.llap.daemon.acl > does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.memory.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.strict.checks.type.safety does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.auto.max.input.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.tez.enable.memory.manager does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.server2.tez.session.lifetime.jitter does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.web.port does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.strict.checks.cartesian.product does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.rpc.num.handlers does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.vcpus.per.instance does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.optimize.partition.columns.separate does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.locality.delay does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.txn.heartbeat.threadpool.size does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.disable.backoff.factor does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.memory.per.instance.mb does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.msck.path.validation does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.reaper.interval does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.tez.input.generate.consistent.splits does not exist > 16/05/04 16:17:16 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.max does not exist > 16/05/04 16:17:18 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/05/04 16:17:18 INFO metastore.HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 16/05/04 16:17:18 INFO metastore.ObjectStore: ObjectStore, initialize called > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateColumns unknown - will be ignored > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateConstraints unknown - will be ignored > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > datanucleus.schema.autoCreateAll unknown - will be ignored > 16/05/04 16:17:18 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateTables unknown - will be ignored > 16/05/04 16:17:18 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 16/05/04 16:17:19 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 16/05/04 16:17:21 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist > 16/05/04 16:17:21 WARN conf.HiveConf: HiveConf of name > hive.llap.io.orc.time.counters does not exist > 16/05/04 16:17:21 WARN conf.HiveConf: HiveConf of name > hive.server2.metrics.enabled does not exist > 16/05/04 16:17:21 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.timeout.ms does not exist > ...... > ( I delete these messages, they're same as above ) > ...... > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.memory.per.instance.mb does not exist > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.msck.path.validation does not exist > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.reaper.interval does not exist > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.tez.input.generate.consistent.splits does not exist > 16/05/04 16:17:22 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.max does not exist > 16/05/04 16:17:22 INFO metastore.ObjectStore: Setting MetaStore object pin > classes with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 16/05/04 16:17:23 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:23 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" > so does not have its own datastore table. > 16/05/04 16:17:25 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:25 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" > so does not have its own datastore table. > 16/05/04 16:17:25 INFO metastore.MetaStoreDirectSql: Using direct SQL, > underlying DB is DERBY > 16/05/04 16:17:25 INFO metastore.ObjectStore: Initialized ObjectStore > 16/05/04 16:17:25 WARN metastore.ObjectStore: Version information not found > in metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 1.2.0 > 16/05/04 16:17:26 WARN metastore.ObjectStore: Failed to get database default, > returning NoSuchObjectException > 16/05/04 16:17:26 INFO metastore.HiveMetaStore: Added admin role in metastore > 16/05/04 16:17:26 INFO metastore.HiveMetaStore: Added public role in metastore > 16/05/04 16:17:26 INFO metastore.HiveMetaStore: No user is added in admin > role, since config is empty > 16/05/04 16:17:26 INFO metastore.HiveMetaStore: 0: get_all_databases > 16/05/04 16:17:26 INFO HiveMetaStore.audit: ugi=yezt ip=unknown-ip-addr > cmd=get_all_databases > 16/05/04 16:17:26 INFO metastore.HiveMetaStore: 0: get_functions: db=default > pat=* > 16/05/04 16:17:26 INFO HiveMetaStore.audit: ugi=yezt ip=unknown-ip-addr > cmd=get_functions: db=default pat=* > 16/05/04 16:17:26 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:27 INFO session.SessionState: Created local directory: > /home/yezt/hive/iotmp > 16/05/04 16:17:28 INFO session.SessionState: Created HDFS directory: > /tmp/hive/yezt/abe15f39-7f47-40ae-87e1-42108c2286ee > 16/05/04 16:17:28 INFO session.SessionState: Created local directory: > /home/yezt/hive/iotmp/abe15f39-7f47-40ae-87e1-42108c2286ee > 16/05/04 16:17:28 INFO session.SessionState: Created HDFS directory: > /tmp/hive/yezt/abe15f39-7f47-40ae-87e1-42108c2286ee/_tmp_space.db > 16/05/04 16:17:28 INFO spark.SparkContext: Running Spark version 1.6.1 > 16/05/04 16:17:28 WARN spark.SparkConf: > SPARK_CLASSPATH was detected (set to > ':/opt/hive/lib/mysql-connector-java-5.1.38-bin.jar'). > This is deprecated in Spark 1.0+. > Please instead use: > - ./spark-submit with --driver-class-path to augment the driver classpath > - spark.executor.extraClassPath to augment the executor classpath > > 16/05/04 16:17:28 WARN spark.SparkConf: Setting > 'spark.executor.extraClassPath' to > ':/opt/hive/lib/mysql-connector-java-5.1.38-bin.jar' as a work-around. > 16/05/04 16:17:28 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' > to ':/opt/hive/lib/mysql-connector-java-5.1.38-bin.jar' as a work-around. > 16/05/04 16:17:28 INFO spark.SecurityManager: Changing view acls to: yezt > 16/05/04 16:17:28 INFO spark.SecurityManager: Changing modify acls to: yezt > 16/05/04 16:17:28 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(yezt); users > with modify permissions: Set(yezt) > 16/05/04 16:17:29 INFO util.Utils: Successfully started service 'sparkDriver' > on port 33667. > 16/05/04 16:17:30 INFO slf4j.Slf4jLogger: Slf4jLogger started > 16/05/04 16:17:30 INFO Remoting: Starting remoting > 16/05/04 16:17:30 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriverActorSystem@192.168.124.129:46614] > 16/05/04 16:17:30 INFO util.Utils: Successfully started service > 'sparkDriverActorSystem' on port 46614. > 16/05/04 16:17:30 INFO spark.SparkEnv: Registering MapOutputTracker > 16/05/04 16:17:30 INFO spark.SparkEnv: Registering BlockManagerMaster > 16/05/04 16:17:30 INFO storage.DiskBlockManager: Created local directory at > /tmp/blockmgr-5933a2ad-a70b-4a2a-98fc-143740ea7721 > 16/05/04 16:17:30 INFO storage.MemoryStore: MemoryStore started with capacity > 517.4 MB > 16/05/04 16:17:30 INFO spark.SparkEnv: Registering OutputCommitCoordinator > 16/05/04 16:17:31 INFO server.Server: jetty-8.y.z-SNAPSHOT > 16/05/04 16:17:31 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 16/05/04 16:17:31 INFO util.Utils: Successfully started service 'SparkUI' on > port 4040. > 16/05/04 16:17:31 INFO ui.SparkUI: Started SparkUI at > http://192.168.124.129:4040 > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Connecting to master > spark://master:7077... > 16/05/04 16:17:31 INFO cluster.SparkDeploySchedulerBackend: Connected to > Spark cluster with app ID app-20160504161731-0003 > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor added: > app-20160504161731-0003/0 on worker-20160504160506-192.168.124.129-52461 > (192.168.124.129:52461) with 1 cores > 16/05/04 16:17:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor > ID app-20160504161731-0003/0 on hostPort 192.168.124.129:52461 with 1 cores, > 1024.0 MB RAM > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor added: > app-20160504161731-0003/1 on worker-20160504160503-192.168.124.130-60681 > (192.168.124.130:60681) with 1 cores > 16/05/04 16:17:31 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40724. > 16/05/04 16:17:31 INFO netty.NettyBlockTransferService: Server created on > 40724 > 16/05/04 16:17:31 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 16/05/04 16:17:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor > ID app-20160504161731-0003/1 on hostPort 192.168.124.130:60681 with 1 cores, > 1024.0 MB RAM > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor added: > app-20160504161731-0003/2 on worker-20160504160504-192.168.124.131-45538 > (192.168.124.131:45538) with 1 cores > 16/05/04 16:17:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor > ID app-20160504161731-0003/2 on hostPort 192.168.124.131:45538 with 1 cores, > 1024.0 MB RAM > 16/05/04 16:17:31 INFO storage.BlockManagerMasterEndpoint: Registering block > manager 192.168.124.129:40724 with 517.4 MB RAM, BlockManagerId(driver, > 192.168.124.129, 40724) > 16/05/04 16:17:31 INFO storage.BlockManagerMaster: Registered BlockManager > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor updated: > app-20160504161731-0003/0 is now RUNNING > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor updated: > app-20160504161731-0003/1 is now RUNNING > 16/05/04 16:17:31 INFO client.AppClient$ClientEndpoint: Executor updated: > app-20160504161731-0003/2 is now RUNNING > 16/05/04 16:17:32 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend > is ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.0 > 16/05/04 16:17:34 INFO hive.HiveContext: Initializing execution hive, version > 1.2.1 > 16/05/04 16:17:34 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0 > 16/05/04 16:17:34 INFO client.ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 > SET hive.support.sql11.reserved.keywords=false > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.io.orc.time.counters does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.server2.metrics.enabled does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.timeout.ms does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.connect.retry.limit does not exist > ...... > ( I delete these messages, they're same as above ) > ...... > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.disable.backoff.factor does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.memory.per.instance.mb does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.msck.path.validation does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.reaper.interval does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.tez.input.generate.consistent.splits does not exist > 16/05/04 16:17:35 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.max does not exist > 16/05/04 16:17:35 INFO hive.HiveContext: default warehouse location is > /user/hive/warehouse > 16/05/04 16:17:35 INFO hive.HiveContext: Initializing HiveMetastoreConnection > version 1.2.1 using Spark classes. > 16/05/04 16:17:35 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0 > 16/05/04 16:17:35 INFO client.ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 > 16/05/04 16:17:36 INFO cluster.SparkDeploySchedulerBackend: Registered > executor NettyRpcEndpointRef(null) (Slave01:59932) with ID 1 > 16/05/04 16:17:36 INFO storage.BlockManagerMasterEndpoint: Registering block > manager Slave01:40271 with 517.4 MB RAM, BlockManagerId(1, Slave01, 40271) > 16/05/04 16:17:36 INFO cluster.SparkDeploySchedulerBackend: Registered > executor NettyRpcEndpointRef(null) (Slave02:39350) with ID 2 > 16/05/04 16:17:36 INFO storage.BlockManagerMasterEndpoint: Registering block > manager Slave02:47549 with 517.4 MB RAM, BlockManagerId(2, Slave02, 47549) > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.io.orc.time.counters does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.server2.metrics.enabled does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.timeout.ms does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.connect.retry.limit does not exist > ...... > ( I delete these messages, they're same as above ) > ...... > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.disable.backoff.factor does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.memory.per.instance.mb does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.msck.path.validation does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.reaper.interval does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.tez.input.generate.consistent.splits does not exist > 16/05/04 16:17:37 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.max does not exist > 16/05/04 16:17:38 INFO metastore.HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 16/05/04 16:17:38 INFO metastore.ObjectStore: ObjectStore, initialize called > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateColumns unknown - will be ignored > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateConstraints unknown - will be ignored > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > datanucleus.schema.autoCreateAll unknown - will be ignored > 16/05/04 16:17:38 INFO DataNucleus.Persistence: Property > datanucleus.schema.validateTables unknown - will be ignored > 16/05/04 16:17:39 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 16/05/04 16:17:39 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.io.orc.time.counters does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.server2.metrics.enabled does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.timeout.ms does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.server2.thrift.client.connect.retry.limit does not exist > ...... > ( I delete these messages, they're same as above ) > ...... > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.task.scheduler.node.disable.backoff.factor does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.am.liveness.connection.sleep.between.retries.ms does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.daemon.memory.per.instance.mb does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.msck.path.validation does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.compactor.history.reaper.interval does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.tez.input.generate.consistent.splits does not exist > 16/05/04 16:17:40 WARN conf.HiveConf: HiveConf of name > hive.llap.io.allocator.alloc.max does not exist > 16/05/04 16:17:40 INFO metastore.ObjectStore: Setting MetaStore object pin > classes with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 16/05/04 16:17:42 INFO cluster.SparkDeploySchedulerBackend: Registered > executor NettyRpcEndpointRef(null) (Master:51517) with ID 0 > 16/05/04 16:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block > manager Master:60869 with 517.4 MB RAM, BlockManagerId(0, Master, 60869) > 16/05/04 16:17:43 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:43 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" > so does not have its own datastore table. > 16/05/04 16:17:43 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:43 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" > so does not have its own datastore table. > 16/05/04 16:17:43 INFO DataNucleus.Query: Reading in results for query > "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is > closing > 16/05/04 16:17:43 INFO metastore.MetaStoreDirectSql: Using direct SQL, > underlying DB is MYSQL > 16/05/04 16:17:43 INFO metastore.ObjectStore: Initialized ObjectStore > 16/05/04 16:17:44 INFO metastore.HiveMetaStore: Added admin role in metastore > 16/05/04 16:17:44 INFO metastore.HiveMetaStore: Added public role in metastore > 16/05/04 16:17:44 INFO metastore.HiveMetaStore: No user is added in admin > role, since config is empty > 16/05/04 16:17:45 INFO metastore.HiveMetaStore: 0: get_all_databases > 16/05/04 16:17:45 INFO HiveMetaStore.audit: ugi=yezt ip=unknown-ip-addr > cmd=get_all_databases > 16/05/04 16:17:45 INFO metastore.HiveMetaStore: 0: get_functions: db=default > pat=* > 16/05/04 16:17:45 INFO HiveMetaStore.audit: ugi=yezt ip=unknown-ip-addr > cmd=get_functions: db=default pat=* > 16/05/04 16:17:45 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as > "embedded-only" so does not have its own datastore table. > 16/05/04 16:17:45 INFO session.SessionState: Created HDFS directory: > /tmp/hive/yezt/a9749204-8d91-436d-8d67-8c2d8f5d524f > 16/05/04 16:17:45 INFO session.SessionState: Created local directory: > /home/yezt/hive/iotmp/a9749204-8d91-436d-8d67-8c2d8f5d524f > 16/05/04 16:17:45 INFO session.SessionState: Created HDFS directory: > /tmp/hive/yezt/a9749204-8d91-436d-8d67-8c2d8f5d524f/_tmp_space.db > SET spark.sql.hive.version=1.2.1 > SET spark.sql.hive.version=1.2.1 > spark-sql> -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org