[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352653#comment-15352653
]
Alexandre Linte commented on HIVE-13830:
----------------------------------------
Hi,
I upgraded Hive to Hive 2.1.0. Now I have the following errors:
- HQL commands
{noformat}
hive (shfs3453)> SELECT COUNT(year) FROM chicagocrimes GROUP BY year;
SELECT COUNT(year) FROM chicagocrimes GROUP BY year
FAILED: SemanticException Failed to get a spark session:
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
{noformat}
- Client logs
{noformat}
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.conf.HiveConfUsing the
default value passed in for log id: c10f51a3-a72d-40c7-9ff6-26e5fb3732da
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.session.SessionStateUpdating thread name to
c10f51a3-a72d-40c7-9ff6-26e5fb3732da main
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.DriverCompiling
command(queryId=shfs3453_20160628110208_f0b51237-d391-472d-abe8-f2dd2457a9ed):
SELECT COUNT(year) FROM chicagocrimes GROUP BY year
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerStarting Semantic Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted phase 1 of Semantic
Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination
tables
Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted getting MetaData in
Semantic Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination
tables
Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.common.FileUtilsCreating
directory if it doesn't exist:
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1/-mr-10001/.hive-staging_hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCBO Succeeded; optimized logical
plan.
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for FS(6)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(5)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(4)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for RS(3)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(2)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(1)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for TS(0)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 oldColExprMap:
{KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 newColExprMap:
{KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryloading spark
properties from:spark-defaults.conf
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.default.parallelism -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryoserializer.buffer -> 100m).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.memory -> 4g).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.driver.memory -> 2g).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryo.classesToRegister ->
org.apache.hadoop.hive.ql.io.HiveKey,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.eventLog.compress -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.akka.frameSize -> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryoserializer.buffer.max -> 1500m).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.sql.hive.metastore.jars -> builtin).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.yarn.scheduler.heartbeat.interval-ms -> 3000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.broadcast.compress -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.eventLog.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.extraClassPath ->
/opt/application/Spark-Cassandra/current/jar/spark-cassandra-connector.jar).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.broadcast.blockSize -> 4096).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.master -> yarn-client).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.io.compression.snappy.blockSize -> 32k).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.cores -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.io.compression.codec ->
org.apache.spark.io.SnappyCompressionCodec).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryo.referenceTracking -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.serializer ->
org.apache.spark.serializer.KryoSerializer).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.instances -> 4).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.extraJavaOptions ->
-Djava.library.path=/opt/application/Hadoop/current/lib/native/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.driver.maxResultSize -> 1200m).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.sql.hive.metastore.version -> 1.2.1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.eventLog.dir -> hdfs:///Products/SPARK/logs/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.leaderport -> 3888).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.localizer.client.thread-count -> 5).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.storefile.refresh.period -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload RPC property
from hive configuration (hive.spark.client.connect.timeout -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.admin.address ->
0.0.0.0:8047).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.property.clientPort -> 2181).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.dfs.client.read.shortcircuit.buffer.size -> 131072).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.failover-retries-on-socket-timeouts -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.thrift.framed.max_frame_size_in_mb -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.address ->
0.0.0.0:0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.ha.automatic-failover.embedded -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.zk-state-store.parent-path -> /yarn/rmstore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.metrics.exposeOperationTimes -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.local.dir -> /tmp/hbase-shfs3453/local/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern ->
^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.client-am.ipc.max-retries -> 3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.keytab.file ->
/opt/application/Hbase/current/keytabs/hbase.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.info.port -> 16030).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.fs.state-store.retry-interval-ms -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (zookeeper.znode.parent -> /hbase).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.regionserver.classes ->
org.apache.hadoop.hbase.security.access.AccessController).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.client.best-effort -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.disk-health-checker.interval-ms -> 120000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.zk-timeout-ms
-> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.aux-services.mapreduce_shuffle.class ->
org.apache.hadoop.mapred.ShuffleHandler).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.health-checker.script.timeout-ms -> 1200000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.client.retry-interval-ms -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.abortonerror -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.bytes.per.checksum -> 16384).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.ha.automatic-failover.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.logroll.errors.tolerated -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.data.umask.enable -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.windows-container.memory-limit.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.root-dir ->
/sharedcache).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.resource-tracker.address.rm1 ->
resourcemanager01.bigdata.fr:8031).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.resource-tracker.address.rm2 ->
resourcemanager02.bigdata.fr:8031).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.webapp.address -> resourcemanager01.bigdata.fr:8188).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.status.publisher.class ->
org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.balancer.period -> 300000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.readonly -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.cluster.distributed -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.app.mapreduce.am.staging-dir
-> /Products/MR/staging).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.env-whitelist ->
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage ->
false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.max.filesize -> 10737418240).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.sleep-delay-before-sigkill.ms -> 250).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.address ->
resourcemanager01.bigdata.fr:10200).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.recovery.enabled
-> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.dns.interface -> default).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.work-preserving-recovery.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.zk-num-retries -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.ttl-ms ->
604800000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.generic-application-history.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.am.liveness-monitor.expiry-interval-ms -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.ipc.client.fallback-to-simple-auth-allowed -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.table.lock.enable -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.property.initLimit -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb -> 1024).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.compactionThreshold -> 3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.webapp.address ->
0.0.0.0:8042).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.webapp.https.address ->
resourcemanager01.bigdata.fr:8190).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.keytab.file ->
/opt/application/Hbase/current/keytabs/hbase.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.admin.address
-> resourcemanager01.bigdata.fr:8033).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.http.filter.initializers ->
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.resource-tracker.address ->
resourcemanager01.bigdata.fr:8031).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.kerberos.principal -> hbase/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.log-aggregation.retain-seconds -> 43200).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.metrics.showTableName -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.nodemanager-client-async.thread-pool-max-size -> 500).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.resource.cpu-vcores -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.status.multicast.address.port -> 16100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.nested-level ->
3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.property.maxClientCnxns -> 300).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.thrift.http -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.aux-services ->
mapreduce_shuffle).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.client.job.max-retries -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs -> 86400).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.fail-fast ->
false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.info.port -> 16010).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.nodemanager-connect.retry-interval-ms -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.container-metrics.unregister-delay-ms -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.write.buffer -> 2097152).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.info.port.auto -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.application-client-protocol.poll-interval-ms -> 200).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.http.max.threads -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.security.qop -> auth).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.app.mapreduce.am.command-opts
-> -Xmx3800m -Xss512k -XX:+UseParallelGC -XX:ParallelGCThreads=4 -verbose:gc
-XX:+PrintTenuringDistribution -XX:+PrintGCDetails -Xloggc:GC.log).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.fail-fast -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.generic-application-history.fs-history-store.uri ->
/Products/YARN/timeline/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.pmem-check-enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.scheduler.minimum-allocation-mb -> 1024).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.http-authentication.type -> kerberos).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.useMulti -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.shuffle.log.backups -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.memstore.block.multiplier -> 4).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.max-completed-applications -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload RPC property
from hive configuration (hive.spark.client.rpc.threads -> 8).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.acl.enable -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nm.liveness-monitor.expiry-interval-ms -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.state-store.max-completed-applications -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.log-aggregation-enable ->
true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.ipc.client.tcpnodelay -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.principal ->
rm/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.storescanner.parallel.seek.enable -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.htablepool.size.max -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.snapshot.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.store.in-memory.staleness-period-mins -> 10080).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.info.bindAddress -> 0.0.0.0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.monitor.policies ->
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.http-authentication.simple.anonymous.allowed -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.local-cache.max-files-per-directory -> 8192).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.admin.acl -> rm,yarn,hdfs).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.cluster-id ->
sandbox-RMS).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.user.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.job.task.listener.thread-count -> 30).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.security.visibility.mutations.checkauth -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.client.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.ha.rm-ids ->
rm1,rm2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.resourcemanager.minimum.version -> NONE).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.cgroups.mount -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.process-kill-wait.ms -> 2000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rootdir -> hdfs://sandbox/Products/HBASE/warehouse/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.zk-auth ->
@/opt/application/Hadoop/current/etc/hadoop/rmhazk-auth.txt).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.server.versionfile.writeattempts -> 3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.server.compactchecker.interval.multiplier -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.ipc.server.callqueue.scan.ratio -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.address.rm2
-> resourcemanager02.bigdata.fr:8032).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.address.rm1
-> resourcemanager01.bigdata.fr:8032).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload RPC property
from hive configuration (hive.spark.client.rpc.max.size -> 52428800).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.job.committer.cancel-timeout -> 60000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.address -> resourcemanager01.bigdata.fr:8030).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.principal ->
nm/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.server.scanner.max.result.size -> 104857600).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.logcleaner.plugins ->
org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size ->
10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.shuffle.log.separate -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds -> -1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.http.policy -> HTTP_ONLY).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.retries.number -> 35).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.authentication.kerberos.principal ->
HTTP/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.catalog.timeout -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.majorcompaction -> 604800000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.localizer.fetch.thread-count -> 4).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.kerberos.principal -> HTTP/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.generic-application-history.max-applications -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.keytab ->
/opt/application/Hadoop/current/keytabs/yarn.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.thrift.compact -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.online.schema.update.enable -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from hive configuration (spark.master -> yarn-client).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.proxy-user-privileges.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.checksum.verify -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.docker-container-executor.exec-name -> /usr/bin/docker).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.lease.recovery.dfs.timeout -> 64000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.fs.tmp.dir -> /user/shfs3453/hbase-staging).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.minWorkerThreads -> 16).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.delayed.delegation-token.removal-interval-ms -> 30000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.catalog.timeout -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.store-class
-> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.app.mapreduce.am.log.level ->
INFO).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.master.classes ->
org.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.delete.thread-count -> 4).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.scheduler.maximum-allocation-vcores -> 24).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.data.umask -> 000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.app.mapreduce.am.resource.mb
-> 4096).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user -> nobody).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.uploader.server.address -> 0.0.0.0:8046).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.infoserver.redirect -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.bin.path ->
/opt/application/Hadoop/current/bin/yarn).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coordinated.state.manager.class ->
org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.preclose.flush.size -> 5242880).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.kerberos.principal -> hbase/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.localityCheck.threadPoolSize -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.localizer.cache.target-size-mb -> 10240).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.port -> 16020).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.status.multicast.address.ip -> 226.1.1.3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.defaults.for.version -> 1.1.2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.remote-app-log-dir -> /Products/YARN/logs).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.bulkload.retries.number -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.hard-kill-timeout-ms -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.scheduler.maximum-allocation-mb -> 65536).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.hostname ->
resourcemanager01.bigdata.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms -> 300000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.region.replica.replication.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.container.liveness-monitor.interval-ms -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.vmem-pmem-ratio
-> 2.1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rpc.shortoperation.timeout -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.hostname ->
0.0.0.0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.cleaner.resource-sleep-ms -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.status.listener.class ->
org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.keyvalue.maxsize -> 10485760).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.pause -> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload RPC property
from hive configuration (hive.spark.client.server.connect.timeout -> 90000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.column.max.version -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.nodes.include-path ->
/opt/application/Hadoop/current/etc/hadoop/slaves).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.status.published -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.cross-origin.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.nodemanager.minimum.version -> NONE).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.port -> 8080).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rpc.timeout -> 60000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.resources-handler.class ->
org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.ttl-enable
-> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size ->
10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.checksum.algo.impl ->
org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.hfilecleaner.plugins ->
org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.cleaner.initial-delay-mins -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.webapp.address ->
0.0.0.0:8788).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.keytab ->
/opt/application/Hadoop/current/keytabs/yarn.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.zk-acl ->
@/opt/application/Hadoop/current/etc/hadoop/rmhazk-acl.txt).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.hlog.reader.impl ->
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.generic-application-history.store-class ->
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
-> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.hlog.writer.impl ->
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.client.job.retry-interval -> 2000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.ha.enabled ->
true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.replication.rpc.codec ->
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.admin.address.rm1 -> resourcemanager01.bigdata.fr:8033).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.admin.address.rm2 -> resourcemanager02.bigdata.fr:8033).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.memstore.flush.size -> 134217728).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.dns.interface -> default).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.log-aggregation.retain-check-interval-seconds -> -1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.msginterval -> 3000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.majorcompaction.jitter -> 0.50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (zookeeper.znode.rootserver -> root-region-server).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.disk-health-checker.min-healthy-disks -> 0.25).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.max-cached-nodemanagers-proxies -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.threads.min -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.resource-tracker.client.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.store.class ->
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.client.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.group -> hadoop).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.resource.memory-mb -> -33088).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.dns.nameserver -> default).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.address.rm1 ->
resourcemanager01.bigdata.fr:8030).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.address.rm2 ->
resourcemanager02.bigdata.fr:8030).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.flusher.count -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.max.total.tasks -> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.kerberos.principal -> stargate/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.handler.count -> 30).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.security.authorization -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.admin.client.thread-count -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.logroll.period -> 3600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.log.retain-seconds -> 10800).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.cgroups.hierarchy -> /hadoop-yarn).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.security.exec.permission.checks -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.container.log.limit.kb -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.admin.thread-count -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regions.slop -> 0.2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.threads.max -> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.shuffle.log.limit.kb -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.snapshot.restore.take.failsafe.snapshot -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.blockingStoreFiles -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts -> 3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.maxQueuedRequests -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.keytab.file -> /etc/krb5/spnego.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.nm.uploader.thread-count -> 20).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.address -> resourcemanager01.bigdata.fr:8088).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.memstore.mslab.enabled -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.remote-app-log-dir-suffix -> logs).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.dns.nameserver -> default).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled ->
false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.security.authentication -> kerberos).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.zk-state-store.root-node.acl -> setindatabag).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.recovery.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.admin-env ->
MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.handler-thread-count -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.hostname ->
0.0.0.0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rootdir.perms -> 700).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.container-monitor.interval-ms -> 3000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.support.proxyuser -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.store.class
-> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rs.cacheblocksonwrite -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.client.max-retries -> 30).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.log-dirs ->
file:///mnt/hd0/hadoop/yarn/log/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.fs.state-store.uri ->
/tmp/hadoop-shfs3453/yarn/system/rmstore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.keytab ->
/opt/application/Hadoop/current/keytabs/yarn.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.security.visibility.mutations.checkauths -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.http-authentication.kerberos.principal ->
HTTP/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-timeline-store.read-cache-size -> 104857600).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.ipc.server.callqueue.read.ratio -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.authentication.kerberos.keytab ->
/etc/krb5/spnego.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.job.committer.commit-window -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.cells.scanned.per.heartbeat.check -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.connect.max-wait.ms -> 900000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.auth.key.update.interval -> 86400000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.config.read.zookeeper.config -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.client.max-retries -> 3).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.defaults.for.version.skip -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.https.address -> 0.0.0.0:8090).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.ipc.rpc.class ->
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.localizer.cache.cleanup.interval-ms -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.amlauncher.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.connect.retry-interval.ms -> 30000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.scheduler.minimum-allocation-vcores -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.dynamic.jars.dir ->
hdfs://sandbox/Products/HBASE/warehouse//lib).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline_service.webapp.address -> resourcemanager01.bigdata.fr:8188).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.nodemanager-connect-retries -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.region.split.policy ->
org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.zk-address ->
namenode01.bigdata.fr:2181,namenode02.bigdata.fr:2181,datanode01.bigdata.fr:2181,).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.store.in-memory.initial-delay-mins -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.resource.percentage-physical-cpu-limit -> 100).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.webapp.cross-origin.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.handler.abort.on.error.percent -> 0.5).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload RPC property
from hive configuration (hive.spark.client.secret.bits -> 256).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.system-metrics-publisher.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.auth.token.max.lifetime -> 604800000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.scanner.timeout.period -> 60000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.port -> 16000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.task.container.log.backups -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.uploader.server.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.class ->
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.keytab.file ->
/opt/application/Hbase/current/keytabs/hbase.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.nodemanager-connect.max-wait-ms -> 180000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.nodes.exclude-path ->
/opt/application/Hadoop/current/etc/hadoop/exclude).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.fs.state-store.retry-policy-spec -> 2000, 500).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.nodemanagers.heartbeat-interval-ms -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.fs.state-store.num-retries -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.compaction.kv.max -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.client.failover-proxy-provider ->
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.quorum ->
namenode01.bigdata.fr:2181,namenode02.bigdata.fr:2181,datanode01.bigdata.fr:2181,).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.snapshot.restore.failsafe.name ->
hbase-failsafe-{snapshot.name}-{restore.timestamp}).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.container-executor.class ->
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.recovery.dir ->
/tmp/hadoop-shfs3453/yarn-nm-recovery).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.scanner.caching -> 2147483647).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.coprocessor.region.classes ->
org.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController,org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.scanner.max.result.size -> 2097152).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.nm.uploader.replication.factor -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.table.max.rowsize -> 1073741824).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.bulkload.staging.dir -> /tmp/hbase-staging).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.lease.recovery.timeout -> 900000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.compaction.max -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.server.thread.wakefrequency -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.thrift.framed -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.recovery.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.address.rm2 -> resourcemanager02.bigdata.fr:8088).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.webapp.address.rm1 -> resourcemanager01.bigdata.fr:8088).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.thrift.maxWorkerThreads -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.dispatcher.drain-events.timeout -> 300000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.superuser -> hdfs).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.time.to.purge.deletes -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs ->
86400).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.client.failover-retries -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.client-server.address -> 0.0.0.0:8045).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.container-manager.thread-count -> 20).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.http.staticuser.user -> dr.stack).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.tmp.dir -> /tmp/hbase-shfs3453).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.sharedcache.app-checker.class
-> org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.container.log.backups -> 0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.property.syncLimit -> 5).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-timeline-store.path ->
/mnt/hd0/hadoop/yarn/timeline).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms -> 10000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.distributed.log.replay -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.support.proxyuser -> true).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.resourcemanager.address ->
resourcemanager01.bigdata.fr:8032).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.windows-container.cpu-limit.enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.localizer.address
-> 0.0.0.0:8040).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.logcleaner.ttl -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.regionSplitLimit -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.client-server.thread-count -> 50).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.health-checker.interval-ms -> 600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.master.loadbalancer.class ->
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.info.bindAddress -> 0.0.0.0).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.scheduler.monitor.enable -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.zk-retry-interval-ms -> 1000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (zookeeper.znode.acl.parent -> acl).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.configuration.provider-class ->
org.apache.hadoop.yarn.LocalConfigurationProvider).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.nodemanager.local-dirs ->
file:///mnt/hd0/hadoop/yarn/local/).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.leveldb-state-store.path ->
/tmp/hadoop-shfs3453/yarn/system/rmstore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.max.perregion.tasks -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.http-authentication.kerberos.keytab ->
/opt/application/Hadoop/current/keytabs/hadoop.keytab).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.state-store-class ->
org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.peerport -> 2888).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.regionserver.optionalcacheflushinterval -> 3600000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.client.max.perserver.tasks -> 5).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.timeline-service.leveldb-state-store.path ->
/tmp/hadoop-shfs3453/yarn/timeline).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.filter.classes ->
org.apache.hadoop.hbase.rest.filter.GzipFilter).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.delete.debug-delay-sec -> 60000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rest.authentication.type -> kerberos).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.rpc.protection -> auth-conf).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.log-aggregation.compression-type -> none).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.vmem-check-enabled -> false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.enabled ->
false).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.zookeeper.property.dataDir ->
/tmp/hbase-shfs3453/zookeeper).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.checksum.algorithm -> CRC32).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.cleaner.period-mins -> 1440).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode (yarn.timeline-service.principal ->
tl/_h...@realm.fr).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hstore.blockingWaitTime -> 90000).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.hregion.percolumnfamilyflush.size.lower.bound -> 16777216).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.ha.automatic-failover.zk-base-path ->
/yarn/leader-election).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.ipc.server.callqueue.handler.factor -> 0.1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.sharedcache.store.in-memory.check-period-mins -> 720).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload HBase
configuration (hbase.storescanner.parallel.seek.threads -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.nodemanager.aux-services.mapreduce.shuffle.class ->
org.apache.hadoop.mapred.ShuffleHandler).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.resourcemanager.am.max-attempts -> 2).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload yarn property
from hive configuration in yarn-client mode
(yarn.app.mapreduce.am.resource.cpu-vcores -> 1).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplLoading spark defaults:
file:/opt/application/Hive/apache-hive-2.1.0-bin/conf/spark-defaults.conf
Jun 28 11:02:08 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplRunning client driver with argv:
kinit hiveserver2/sparkclient01.bigdata...@realm.fr -k -t
/opt/application/Hive/current/keytabs/hiveserver2.keytab;
/opt/application/Spark/current/bin/spark-submit --properties-file
/tmp/spark-submit.1914700103794483224.properties --class
org.apache.hive.spark.client.RemoteDriver
/opt/application/Hive/apache-hive-2.1.0-bin/lib/hive-exec-2.1.0.jar
--remote-host sparkclient01.bigdata.fr --remote-port 64022 --conf
hive.spark.client.connect.timeout=1000 --conf
hive.spark.client.server.connect.timeout=90000 --conf
hive.spark.client.channel.log.level=null --conf
hive.spark.client.rpc.max.size=52428800 --conf hive.spark.client.rpc.threads=8
--conf hive.spark.client.secret.bits=256 --conf
hive.spark.client.rpc.server.address=null
Jun 28 11:02:08 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplkinit: Client
'hiveserver2/sparkclient01.bigdata...@realm.fr' not found in Kerberos database
while getting initial credentials
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplWarning: Ignoring non-spark config
property: hive.spark.client.server.connect.timeout=90000
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplWarning: Ignoring non-spark config
property: hive.spark.client.rpc.threads=8
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplWarning: Ignoring non-spark config
property: hive.spark.client.connect.timeout=1000
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplWarning: Ignoring non-spark config
property: hive.spark.client.secret.bits=256
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplWarning: Ignoring non-spark config
property: hive.spark.client.rpc.max.size=52428800
Jun 28 11:02:10 hive-cli INFO -
org.apache.hive.spark.client.SparkClientImplException in thread "main"
java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at
org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:134)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:516)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at java.lang.reflect.Method.invoke(Method.java:606)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
Jun 28 11:02:10 hive-cli INFO - org.apache.hive.spark.client.SparkClientImpl
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Jun 28 11:02:10 hive-cli ERROR -
org.apache.hive.spark.client.SparkClientImplError while waiting for client to
connect.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancel
client '9f709de7-d5c2-4a16-9346-40568e651542'. Error: Child process exited
before connecting back with error log kinit: Client
'hiveserver2/sparkclient01.bigdata...@realm.fr' not found in Kerberos database
while getting initial credentials
Warning: Ignoring non-spark config property:
hive.spark.client.server.connect.timeout=90000
Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
Warning: Ignoring non-spark config property:
hive.spark.client.connect.timeout=1000
Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
Warning: Ignoring non-spark config property:
hive.spark.client.rpc.max.size=52428800
Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
at
org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
at
org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:134)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:516)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:104)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:99)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:95)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:67)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:114)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136)
at
org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:117)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:178)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:116)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:134)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10857)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:239)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:329)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1158)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1253)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Cancel client
'9f709de7-d5c2-4a16-9346-40568e651542'. Error: Child process exited before
connecting back with error log kinit: Client
'hiveserver2/sparkclient01.bigdata...@realm.fr' not found in Kerberos database
while getting initial credentials
Warning: Ignoring non-spark config property:
hive.spark.client.server.connect.timeout=90000
Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
Warning: Ignoring non-spark config property:
hive.spark.client.connect.timeout=1000
Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
Warning: Ignoring non-spark config property:
hive.spark.client.rpc.max.size=52428800
Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
at
org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
at
org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:134)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:516)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at
org.apache.hive.spark.client.rpc.RpcServer.cancelClient(RpcServer.java:179)
at
org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:465)
at java.lang.Thread.run(Thread.java:745)
Jun 28 11:02:10 hive-cli WARN -
org.apache.hive.spark.client.SparkClientImplChild process exited with code 1
Jun 28 11:02:10 hive-cli ERROR - org.apache.hadoop.hive.ql.DriverFAILED:
SemanticException Failed to get a spark session:
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark
session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
spark client.
at
org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:121)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:178)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:116)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:134)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10857)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:239)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:329)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1158)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1253)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Jun 28 11:02:10 hive-cli INFO - org.apache.hadoop.hive.ql.DriverCompleted
compiling
command(queryId=shfs3453_20160628110208_f0b51237-d391-472d-abe8-f2dd2457a9ed);
Time taken: 2.496 seconds
Jun 28 11:02:10 hive-cli INFO - org.apache.hadoop.hive.conf.HiveConfUsing the
default value passed in for log id: c10f51a3-a72d-40c7-9ff6-26e5fb3732da
Jun 28 11:02:10 hive-cli INFO -
org.apache.hadoop.hive.ql.session.SessionStateResetting thread name to main
{noformat}
It's very strange that Hive tries to do a kinit on the machine, with a wrong
principal.
> Hive on spark driver crash with Spark 1.6.1
> -------------------------------------------
>
> Key: HIVE-13830
> URL: https://issues.apache.org/jira/browse/HIVE-13830
> Project: Hive
> Issue Type: Bug
> Components: Spark, spark-branch
> Affects Versions: 2.0.0, 2.1.0
> Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos
> Reporter: Alexandre Linte
>
> With Hive 1.2.1 I was able to use Hive on Spark successfully with the use of
> the spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar".
> Today with Hive 2.0.0, I'm unable to use Hive on Spark whether it be with the
> spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar" or the spark-assembly
> "spark-assembly-1.6.1-hadoop2.7.2.jar".
> My configuration is the following:
> * spark-default.conf available in HIVE_DIR/conf
> * spark assembly available in HIVE_DIR/lib
> I gathered several logs below:
> - HQL commands
> {noformat}
> $ hive -v --database shfs3453
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Spark/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Logging initialized using configuration in
> file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties
> use shfs3453
> OK
> Time taken: 1.425 seconds
> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
> versions. Consider using a different execution engine (i.e. tez, spark) or
> using Hive 1.X releases.
> hive (shfs3453)> set hive.execution.engine=spark;
> set hive.execution.engine=spark
> hive (shfs3453)> set spark.master=yarn-client;
> set spark.master=yarn-client
> hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING,
> Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING,
> LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT,
> District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT,
> YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude
> FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED
> AS TEXTFILE;
> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block
> STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription
> STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT,
> CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year
> INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE
> OK
> Time taken: 0.408 seconds
> hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM
> chicagocrimes WHERE Description = 'FIRST DEGREE MURDER';
> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE
> Description = 'FIRST DEGREE MURDER'
> Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Starting Spark Job = 79484279-8e75-4b13-8e71-7de463f4d51e
> Status: SENT
> Failed to execute spark task, with exception
> 'java.lang.IllegalStateException(RPC channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> - Client logs
> {noformat}
> May 24 09:32:19 hive-cli WARN -
> org.apache.hive.spark.client.rpc.RpcDispatcherReceived error
> message:io.netty.handler.codec.DecoderException:
> java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
> at
> io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:278)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hive.spark.client.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:97)
> at
> io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.spark.client.Job
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 39 more
> .
> May 24 09:32:19 hive-cli WARN -
> org.apache.hive.spark.client.SparkClientImplClient RPC channel closed
> unexpectedly.
> May 24 09:32:48 hive-cli INFO -
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitorJob hasn't been
> submitted after 61s. Aborting it.
> May 24 09:32:48 hive-cli ERROR -
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitorStatus: SENT
> May 24 09:32:48 hive-cli ERROR -
> org.apache.hadoop.hive.ql.exec.spark.SparkTaskFailed to execute spark task,
> with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
> java.lang.IllegalStateException: RPC channel is closed.
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
> at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
> at
> org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:523)
> at
> org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:187)
> at
> org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:128)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:778)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> May 24 09:32:48 hive-cli ERROR -
> org.apache.hadoop.hive.ql.exec.spark.SparkTaskFailed to execute spark task,
> with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
> java.lang.IllegalStateException: RPC channel is closed.
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> ~[guava-14.0.1.jar:?]
> at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:523)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:187)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
> ~[hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:128)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> [hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> [hive-exec-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
> [hive-exec-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> [hive-cli-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> [hive-cli-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
> [hive-cli-2.0.0.jar:2.0.0]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:778)
> [hive-cli-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717)
> [hive-cli-2.0.0.jar:2.0.0]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> [hive-cli-2.0.0.jar:2.0.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.7.0_85]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[?:1.7.0_85]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_85]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_85]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> [spark-assembly-1.6.1-hadoop2.7.2.jar:1.6.1]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> [spark-assembly-1.6.1-hadoop2.7.2.jar:1.6.1]
> May 24 09:32:48 hive-cli ERROR - org.apache.hadoop.hive.ql.DriverFAILED:
> Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> May 24 09:32:48 hive-cli INFO - org.apache.hadoop.hive.ql.DriverCompleted
> executing
> command(queryId=shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73);
> Time taken: 65.543 seconds
> {noformat}
> - Yarn logs (executor)
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/mnt/hd4/hadoop/yarn/local/usercache/shfs3453/filecache/18/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 16/05/24 09:32:14 INFO executor.CoarseGrainedExecutorBackend: Registered
> signal handlers for [TERM, HUP, INT]
> 16/05/24 09:32:15 INFO spark.SecurityManager: Changing view acls to: shfs3453
> 16/05/24 09:32:15 INFO spark.SecurityManager: Changing modify acls to:
> shfs3453
> 16/05/24 09:32:15 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(shfs3453); users
> with modify permissions: Set(shfs3453)
> 16/05/24 09:32:15 INFO spark.SecurityManager: Changing view acls to: shfs3453
> 16/05/24 09:32:15 INFO spark.SecurityManager: Changing modify acls to:
> shfs3453
> 16/05/24 09:32:15 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(shfs3453); users
> with modify permissions: Set(shfs3453)
> 16/05/24 09:32:16 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 16/05/24 09:32:16 INFO Remoting: Starting remoting
> 16/05/24 09:32:16 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkexecutoractorsys...@datanode05.bigdata.fr:37444]
> 16/05/24 09:32:16 INFO util.Utils: Successfully started service
> 'sparkExecutorActorSystem' on port 37444.
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd8/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-8206d302-c8c7-4c79-974f-e349586c64f3
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd3/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-c3f44368-0b14-4bbf-b216-9fb332fd5174
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd2/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-008678f2-f592-4026-8342-9f03432539bc
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd1/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-44aad90f-755c-4294-b294-355b929b43e7
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd9/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-0862fb2d-aa29-48eb-abde-e4b8fa6943af
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd10/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-10811a27-1cb0-44cb-a842-db98638d67b5
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd4/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-b6f7eb6f-bb1e-4f18-80a8-d0e0de4c0c5e
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd5/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-c0949d83-1c03-457d-8c09-b61ae02c9567
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd6/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-c33365d4-9ff0-41a9-9684-19fbd3a8c8ec
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd0/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-94c7ebea-3df2-4004-911d-03e46f02909d
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd11/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-316bc7e1-1cb6-4a68-bf64-f4169d433e4e
> 16/05/24 09:32:17 INFO storage.DiskBlockManager: Created local directory at
> /mnt/hd7/hadoop/yarn/local/usercache/shfs3453/appcache/application_1463151644662_0023/blockmgr-d4b6e343-2f03-4558-acc6-027b896491ee
> 16/05/24 09:32:17 INFO storage.MemoryStore: MemoryStore started with capacity
> 2.7 GB
> 16/05/24 09:32:17 INFO executor.CoarseGrainedExecutorBackend: Connecting to
> driver: spark://CoarseGrainedScheduler@192.168.200.208:55898
> 16/05/24 09:32:17 INFO executor.CoarseGrainedExecutorBackend: Successfully
> registered with driver
> 16/05/24 09:32:17 INFO executor.Executor: Starting executor ID 4 on host
> datanode05.bigdata.fr
> 16/05/24 09:32:17 INFO util.Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49034.
> 16/05/24 09:32:17 INFO netty.NettyBlockTransferService: Server created on
> 49034
> 16/05/24 09:32:17 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 16/05/24 09:32:17 INFO storage.BlockManagerMaster: Registered BlockManager
> 16/05/24 09:32:19 INFO executor.CoarseGrainedExecutorBackend: Driver
> commanded a shutdown
> 16/05/24 09:32:20 INFO storage.MemoryStore: MemoryStore cleared
> 16/05/24 09:32:20 INFO storage.BlockManager: BlockManager stopped
> 16/05/24 09:32:20 WARN executor.CoarseGrainedExecutorBackend: An unknown
> (hiveclient.bigdata.fr:55898) driver disconnected.
> 16/05/24 09:32:20 ERROR executor.CoarseGrainedExecutorBackend: Driver
> 192.168.200.208:55898 disassociated! Shutting down.
> 16/05/24 09:32:20 INFO util.ShutdownHookManager: Shutdown hook called
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
