Hello team,
I am trying to use spark as the engine for Hive.
Hive version : 3.1.1
Spark: 2.4.0 (have tried with several versions)
I am able to query the hive tables from within pyspark.However, when i
execute the hive query from the hive shell, it throws the following error
Launch Command: "/opt/CDS/pre_requisites/jdk1.8.0_141/bin/java" "-cp"
"/opt/CDS/spark/conf/:/opt/CDS/spark/jars/*:/opt/CDS/hadoop-2.9.2/etc/hadoop/:/etc/tez/conf:/opt/CDS/hive/conf/:/opt/CDS/hive/lib/HikariCP-2.6.1.jar:/opt/CDS/hive/lib/ST4-4.0.4.jar:/opt/CDS/hive/lib/accumulo-core-1.7.3.jar:/opt/CDS/hive/lib/accumulo-fate-1.7.3.jar:/opt/CDS/hive/lib/accumulo-start-1.7.3.jar:/opt/CDS/hive/lib/accumulo-trace-1.7.3.jar:/opt/CDS/hive/lib/aircompressor-0.10.jar:/opt/CDS/hive/lib/ant-1.9.1.jar:/opt/CDS/hive/lib/ant-launcher-1.9.1.jar:/opt/CDS/hive/lib/antlr-runtime-3.5.2.jar:/opt/CDS/hive/lib/antlr4-runtime-4.5.jar:/opt/CDS/hive/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/CDS/hive/lib/apache-jsp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/apache-jstl-9.3.20.v20170531.jar:/opt/CDS/hive/lib/arrow-format-0.8.0.jar:/opt/CDS/hive/lib/arrow-memory-0.8.0.jar:/opt/CDS/hive/lib/arrow-vector-0.8.0.jar:/opt/CDS/hive/lib/asm-5.0.1.jar:/opt/CDS/hive/lib/asm-commons-5.0.1.jar:/opt/CDS/hive/lib/asm-tree-5.0.1.jar:/opt/CDS/hive/lib/audience-annotations-0.5.0.jar:/opt/CDS/hive/lib/avatica-1.11.0.jar:/opt/CDS/hive/lib/avro-1.7.7.jar:/opt/CDS/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/CDS/hive/lib/calcite-core-1.16.0.jar:/opt/CDS/hive/lib/calcite-druid-1.16.0.jar:/opt/CDS/hive/lib/calcite-linq4j-1.16.0.jar:/opt/CDS/hive/lib/commons-cli-1.2.jar:/opt/CDS/hive/lib/commons-codec-1.7.jar:/opt/CDS/hive/lib/commons-collections4-4.1.jar:/opt/CDS/hive/lib/commons-compiler-2.7.6.jar:/opt/CDS/hive/lib/commons-compress-1.9.jar:/opt/CDS/hive/lib/commons-crypto-1.0.0.jar:/opt/CDS/hive/lib/commons-dbcp-1.4.jar:/opt/CDS/hive/lib/commons-io-2.4.jar:/opt/CDS/hive/lib/commons-lang-2.6.jar:/opt/CDS/hive/lib/commons-lang3-3.2.jar:/opt/CDS/hive/lib/commons-logging-1.0.4.jar:/opt/CDS/hive/lib/commons-math-2.1.jar:/opt/CDS/hive/lib/commons-math3-3.6.1.jar:/opt/CDS/hive/lib/commons-pool-1.5.4.jar:/opt/CDS/hive/lib/commons-vfs2-2.1.jar:/opt/CDS/hive/lib/curator-client-2.12.0.jar:/opt/CDS/hive/lib/curator-framework-2.12.0.jar:/opt/CDS/hive/lib/curator-recipes-2.12.0.jar:/opt/CDS/hive/lib/datanucleus-api-jdo-4.2.4.jar:/opt/CDS/hive/lib/datanucleus-core-4.1.17.jar:/opt/CDS/hive/lib/datanucleus-rdbms-4.1.19.jar:/opt/CDS/hive/lib/derby-10.14.1.0.jar:/opt/CDS/hive/lib/disruptor-3.3.6.jar:/opt/CDS/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/CDS/hive/lib/druid-hdfs-storage-0.12.0.jar:/opt/CDS/hive/lib/ecj-4.4.2.jar:/opt/CDS/hive/lib/esri-geometry-api-2.0.0.jar:/opt/CDS/hive/lib/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hive/lib/flatbuffers-1.2.0-3f79e055.jar:/opt/CDS/hive/lib/groovy-all-2.4.11.jar:/opt/CDS/hive/lib/gson-2.2.4.jar:/opt/CDS/hive/lib/guava-19.0.jar:/opt/CDS/hive/lib/hbase-client-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-http-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-mapreduce-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-api-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-prefix-tree-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-procedure-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-shaded-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-replication-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-server-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-shaded-miscellaneous-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-netty-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-protobuf-1.0.1.jar:/opt/CDS/hive/lib/hive-accumulo-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-beeline-3.1.1.jar:/opt/CDS/hive/lib/hive-classification-3.1.1.jar:/opt/CDS/hive/lib/hive-cli-3.1.1.jar:/opt/CDS/hive/lib/hive-common-3.1.1.jar:/opt/CDS/hive/lib/hive-contrib-3.1.1.jar:/opt/CDS/hive/lib/hive-druid-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-exec-3.1.1.jar:/opt/CDS/hive/lib/hive-hbase-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-core-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-server-extensions-3.1.1.jar:/opt/CDS/hive/lib/hive-hplsql-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-kryo-registrator-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1-tests.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-ext-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-server-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-tez-3.1.1.jar:/opt/CDS/hive/lib/hive-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-serde-3.1.1.jar:/opt/CDS/hive/lib/hive-service-3.1.1.jar:/opt/CDS/hive/lib/hive-service-rpc-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-0.23-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-common-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-scheduler-3.1.1.jar:/opt/CDS/hive/lib/hive-standalone-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-storage-api-2.7.0.jar:/opt/CDS/hive/lib/hive-streaming-3.1.1.jar:/opt/CDS/hive/lib/hive-testutils-3.1.1.jar:/opt/CDS/hive/lib/hive-upgrade-acid-3.1.1.jar:/opt/CDS/hive/lib/hive-vector-code-gen-3.1.1.jar:/opt/CDS/hive/lib/hk2-api-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-locator-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-utils-2.5.0-b32.jar:/opt/CDS/hive/lib/hppc-0.7.2.jar:/opt/CDS/hive/lib/htrace-core-3.2.0-incubating.jar:/opt/CDS/hive/lib/httpclient-4.5.2.jar:/opt/CDS/hive/lib/httpcore-4.4.4.jar:/opt/CDS/hive/lib/ivy-2.4.0.jar:/opt/CDS/hive/lib/jackson-annotations-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-asl-1.9.13.jar:/opt/CDS/hive/lib/jackson-databind-2.9.5.jar:/opt/CDS/hive/lib/jackson-dataformat-smile-2.9.5.jar:/opt/CDS/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/CDS/hive/lib/jamon-runtime-2.3.1.jar:/opt/CDS/hive/lib/janino-2.7.6.jar:/opt/CDS/hive/lib/javassist-3.20.0-GA.jar:/opt/CDS/hive/lib/javax.annotation-api-1.2.jar:/opt/CDS/hive/lib/javax.inject-2.5.0-b32.jar:/opt/CDS/hive/lib/javax.jdo-3.2.0-m3.jar:/opt/CDS/hive/lib/javax.servlet-api-3.1.0.jar:/opt/CDS/hive/lib/javax.servlet.jsp-2.3.2.jar:/opt/CDS/hive/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/CDS/hive/lib/javax.ws.rs-api-2.0.1.jar:/opt/CDS/hive/lib/javolution-5.5.1.jar:/opt/CDS/hive/lib/jcodings-1.0.18.jar:/opt/CDS/hive/lib/jcommander-1.32.jar:/opt/CDS/hive/lib/jdo-api-3.0.1.jar:/opt/CDS/hive/lib/jersey-client-2.25.1.jar:/opt/CDS/hive/lib/jersey-common-2.25.1.jar:/opt/CDS/hive/lib/jersey-container-servlet-core-2.25.1.jar:/opt/CDS/hive/lib/jersey-guava-2.25.1.jar:/opt/CDS/hive/lib/jersey-media-jaxb-2.25.1.jar:/opt/CDS/hive/lib/jersey-server-2.25.1.jar:/opt/CDS/hive/lib/jettison-1.1.jar:/opt/CDS/hive/lib/jetty-annotations-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-http-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-io-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jaas-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jndi-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-plus-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-rewrite-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-runner-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-schemas-3.1.jar:/opt/CDS/hive/lib/jetty-security-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-util-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-webapp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-xml-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jline-2.12.jar:/opt/CDS/hive/lib/joda-time-2.9.9.jar:/opt/CDS/hive/lib/joni-2.1.11.jar:/opt/CDS/hive/lib/jpam-1.1.jar:/opt/CDS/hive/lib/json-1.8.jar:/opt/CDS/hive/lib/jsr305-3.0.0.jar:/opt/CDS/hive/lib/jta-1.1.jar:/opt/CDS/hive/lib/kryo-shaded-4.0.2.jar:/opt/CDS/hive/lib/libfb303-0.9.3.jar:/opt/CDS/hive/lib/libthrift-0.9.3.jar:/opt/CDS/hive/lib/memory-0.9.0.jar:/opt/CDS/hive/lib/metrics-core-3.1.0.jar:/opt/CDS/hive/lib/metrics-json-3.1.0.jar:/opt/CDS/hive/lib/metrics-jvm-3.1.0.jar:/opt/CDS/hive/lib/minlog-1.3.0.jar:/opt/CDS/hive/lib/mysql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/netty-3.10.5.Final.jar:/opt/CDS/hive/lib/netty-all-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-buffer-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-common-4.1.17.Final.jar:/opt/CDS/hive/lib/opencsv-2.3.jar:/opt/CDS/hive/lib/orc-core-1.5.1.jar:/opt/CDS/hive/lib/orc-shims-1.5.1.jar:/opt/CDS/hive/lib/org.abego.treelayout.core-1.0.1.jar:/opt/CDS/hive/lib/osgi-resource-locator-1.0.1.jar:/opt/CDS/hive/lib/paranamer-2.3.jar:/opt/CDS/hive/lib/parquet-hadoop-bundle-1.10.0.jar:/opt/CDS/hive/lib/postgresql-9.4.1208.jre7.jar:/opt/CDS/hive/lib/postgresql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/protobuf-java-2.5.0.jar:/opt/CDS/hive/lib/scala-library-2.11.12.jar:/opt/CDS/hive/lib/sketches-core-0.9.0.jar:/opt/CDS/hive/lib/snappy-java-1.1.4.jar:/opt/CDS/hive/lib/spark-core_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-network-common_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-unsafe_2.11-2.4.0.jar:/opt/CDS/hive/lib/sqlline-1.3.0.jar:/opt/CDS/hive/lib/stax-api-1.0.1.jar:/opt/CDS/hive/lib/super-csv-2.2.0.jar:/opt/CDS/hive/lib/taglibs-standard-impl-1.2.5.jar:/opt/CDS/hive/lib/taglibs-standard-spec-1.2.5.jar:/opt/CDS/hive/lib/tempus-fugit-1.1.jar:/opt/CDS/hive/lib/transaction-api-1.1.jar:/opt/CDS/hive/lib/validation-api-1.1.0.Final.jar:/opt/CDS/hive/lib/velocity-1.5.jar:/opt/CDS/hive/lib/websocket-api-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-common-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/zookeeper-3.4.6.jar:/opt/CDS/hadoop-2.9.2/share/hadoop/tools/lib/hadoop-distcp-2.9.2.jar:/opt/CDS/hbase//conf/:/opt/CDS/hbase//lib/shaded-clients/hbase-shaded-mapreduce-2.1.1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/commons-logging-1.2.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/CDS/hive/lib/log4j-1.2-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-core-2.10.0.jar:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar:/opt/CDS/hive/lib/log4j-web-2.10.0.jar:/opt/CDS/hadoop-2.9.2/contrib/capacity-scheduler/*.jar:/opt/CDS/hadoop-2.9.2//share/hadoop/common/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/common/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/lib/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/*"
"-Xmx4096M"
"-Dspark.hadoop.hbase.http.filter.initializers=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter"
"-Dspark.hadoop.hbase.defaults.for.version=2.0.0-alpha4"
"-Dspark.hadoop.hbase.wal.dir.perms=700"
"-Dspark.hadoop.hbase.hstore.checksum.algorithm=CRC32C"
"-Dspark.hadoop.hbase.regionserver.thrift.framed.max_frame_size_in_mb=2"
"-Dspark.hadoop.hbase.snapshot.restore.take.failsafe.snapshot=true"
"-Dspark.hadoop.hbase.regionserver.port=60020"
"-Dspark.hadoop.hbase.rpc.rows.warning.threshold=5000"
"-Dspark.hadoop.hbase.storescanner.parallel.seek.enable=false"
"-Dspark.hadoop.hbase.client.max.perregion.tasks=1"
"-Dspark.hadoop.hbase.mob.compaction.threads.max=1"
"-Dspark.hadoop.hbase.mob.compaction.mergeable.threshold=1342177280"
"-Dspark.hadoop.hbase.rpc.timeout=60000"
"-Dspark.hadoop.hbase.replication.rpc.codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags"
"-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"
"-Dspark.hadoop.hbase.auth.token.max.lifetime=604800000"
"-Dspark.hadoop.hbase.mob.compaction.chore.period=604800"
"-Dspark.hadoop.hbase.bulkload.retries.number=10"
"-Dspark.hadoop.hbase.hregion.memstore.flush.size=134217728"
"-Dspark.master=spark://172.30.61.226:7077"
"-Dspark.hadoop.hbase.master.port=60000"
"-Dspark.hadoop.hbase.server.versionfile.writeattempts=3"
"-Dspark.hadoop.hbase.regionserver.logroll.errors.tolerated=2"
"-Dspark.hadoop.hbase.unsafe.stream.capability.enforce=false"
"-Dspark.hadoop.hbase.master.info.port=60010" "-Dspark.app.name=Hive on
Spark (sessionId = 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)"
"-Dspark.hadoop.hbase.table.lock.enable=true"
"-Dspark.hadoop.hbase.lease.recovery.dfs.timeout=64000"
"-Dspark.hadoop.hbase.regionserver.thrift.compact=false"
"-Dspark.hadoop.hbase.hregion.majorcompaction=604800000"
"-Dspark.hadoop.hbase.zookeeper.peerport=2888"
"-Dspark.hadoop.hbase.cluster.distributed=true"
"-Dspark.hadoop.hbase.regionserver.storefile.refresh.period=0"
"-Dspark.kryo.classesToRegister=org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,org.apache.hadoop.io.Writable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.io.HiveKey"
"-Dspark.hadoop.hbase.regionserver.optionalcacheflushinterval=3600000"
"-Dspark.hadoop.hbase.master.procedurewalcleaner.ttl=604800000"
"-Dspark.hadoop.hbase.zookeeper.quorum=172.30.61.229,172.30.61.230,172.30.61.231"
"-Dspark.sql.warehouse.dir=/hive/cdswarehouse"
"-Dspark.hadoop.fs.defaultFS=hdfs://cdscluster/"
"-Dspark.hadoop.hbase.hstore.blockingStoreFiles=10"
"-Dspark.hadoop.hbase.regionserver.hlog.reader.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader"
"-Dspark.hadoop.hbase.regionserver.checksum.verify=true"
"-Dspark.hadoop.hbase.rest.readonly=false"
"-Dspark.hadoop.hbase.regionserver.majorcompaction.pagecache.drop=true"
"-Dspark.submit.deployMode=cluster"
"-Dspark.hadoop.hbase.http.max.threads=16"
"-Dspark.hadoop.hbase.data.umask.enable=false"
"-Dspark.hadoop.hbase.hstore.flusher.count=2"
"-Dspark.hadoop.hbase.server.compactchecker.interval.multiplier=1000"
"-Dspark.hadoop.hbase.hstore.compaction.max.size=9223372036854775807"
"-Dspark.hadoop.hbase.rest.filter.classes=org.apache.hadoop.hbase.rest.filter.GzipFilter"
"-Dspark.hadoop.hbase.rest.csrf.enabled=false"
"-Dspark.hadoop.hbase.coordinated.state.manager.class=org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager"
"-Dspark.hadoop.hbase.mob.delfile.max.count=3"
"-Dspark.hadoop.hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer"
"-Dspark.hadoop.hbase.snapshot.region.timeout=300000"
"-Dspark.hadoop.hbase.hstore.compactionThreshold=3"
"-Dspark.hadoop.hbase.data.umask=000"
"-Dspark.client.authentication.secret=0ffffffea690fffffffb4d0ffffffc045240ffffffff410ffffffd56874181d04700ffffffae320ffffffc40ffffffe036c1a321c63e5e0ffffffde213a3d"
"-Dspark.hadoop.hbase.hstore.bytes.per.checksum=16384"
"-Dspark.hadoop.hbase.server.scanner.max.result.size=104857600"
"-Dspark.hadoop.hbase.ipc.server.callqueue.scan.ratio=0"
"-Dspark.hadoop.hbase.ipc.client.tcpnodelay=true"
"-Dspark.hadoop.hbase.hstore.compaction.ratio.offpeak=5.0F"
"-Dspark.hadoop.hbase.zookeeper.property.dataDir=/data/CDS/zookeeper"
"-Dspark.jars=file:/opt/CDS/hive/lib/hive-exec-3.1.1.jar"
"-Dspark.hadoop.hbase.client.pause=100"
"-Dspark.hadoop.hbase.coprocessor.enabled=true"
"-Dspark.hadoop.hbase.thrift.minWorkerThreads=16"
"-Dspark.kryo.referenceTracking=false"
"-Dspark.hadoop.hbase.master.mob.ttl.cleaner.period=86400"
"-Dspark.hadoop.hbase.status.published=false"
"-Dspark.hadoop.hbase.client.retries.number=35"
"-Dspark.hadoop.hbase.hstore.blockingWaitTime=90000"
"-Dspark.hadoop.hbase.mob.cache.evict.remain.ratio=0.5f"
"-Dspark.hadoop.hbase.master.normalizer.class=org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer"
"-Dspark.hadoop.hbase.client.scanner.caching=2147483647"
"-Dspark.hadoop.hbase.normalizer.period=300000"
"-Dspark.hadoop.hbase.hregion.max.filesize=10737418240"
"-Dspark.hadoop.hbase.hregion.memstore.mslab.enabled=true"
"-Dspark.hadoop.hbase.hstore.compaction.ratio=1.2F"
"-Dspark.hadoop.hbase.security.visibility.mutations.checkauths=false"
"-Dspark.hadoop.hbase.client.localityCheck.threadPoolSize=2"
"-Dspark.hadoop.hbase.master.hfilecleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner"
"-Dspark.hadoop.hbase.coprocessor.user.enabled=true"
"-Dspark.hadoop.hbase.rs.cacheblocksonwrite=false"
"-Dspark.executor.memory=6048m"
"-Dspark.hadoop.hbase.regionserver.msginterval=3000"
"-Dspark.hadoop.hbase.snapshot.master.timeout.millis=300000"
"-Dspark.executor.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
"-Dspark.hadoop.cloneConf=true"
"-Dspark.hadoop.hbase.fs.tmp.dir=/user/cdsuser/hbase-staging"
"-Dspark.hadoop.hbase.mob.cache.evict.period=3600"
"-Dspark.hadoop.hbase.security.exec.permission.checks=false"
"-Dspark.hadoop.hbase.hregion.percolumnfamilyflush.size.lower.bound.min=16777216"
"-Dspark.hadoop.hbase.zookeeper.property.syncLimit=5"
"-Dspark.history.fs.logDirectory=hdfs://cdscluster/spark-logs"
"-Dspark.hadoop.hbase.client.operation.timeout=1200000"
"-Dspark.hadoop.hbase.ipc.server.callqueue.handler.factor=0.1"
"-Dspark.hadoop.hbase.rootdir=hdfs://cdscluster/hbase"
"-Dspark.hadoop.hbase.ipc.server.callqueue.read.ratio=0"
"-Dspark.eventLog.enabled=true"
"-Dspark.hadoop.hbase.regionserver.handler.abort.on.error.percent=0.5"
"-Dspark.hadoop.hbase.thrift.maxWorkerThreads=1000"
"-Dspark.hadoop.hbase.regionserver.logroll.period=3600000"
"-Dspark.hadoop.hbase.mob.file.cache.size=1000"
"-Dspark.hadoop.hbase.master.logcleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner"
"-Dspark.hadoop.hbase.coprocessor.abortonerror=true"
"-Dspark.hadoop.hbase.hregion.preclose.flush.size=5242880"
"-Dspark.hadoop.hbase.dynamic.jars.dir=hdfs://cdscluster/hbase/lib"
"-Dspark.hadoop.hbase.column.max.version=1"
"-Dspark.hadoop.hbase.master.logcleaner.ttl=600000"
"-Dspark.hadoop.hbase.regionserver.hostname.disable.master.reversedns=false"
"-Dspark.hadoop.hbase.replication.source.maxthreads=10"
"-Dspark.hadoop.hbase.client.scanner.timeout.period=60000"
"-Dspark.hadoop.hbase.security.authentication=simple"
"-Dspark.hadoop.hbase.snapshot.enabled=true"
"-Dspark.hadoop.hbase.master.balancer.maxRitPercent=1.0"
"-Dspark.hadoop.hbase.zookeeper.dns.interface=default"
"-Dspark.driver.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
"-Dspark.hadoop.hbase.display.keys=true"
"-Dspark.hadoop.hbase.regionserver.info.port.auto=false"
"-Dspark.hadoop.hbase.client.write.buffer=2097152"
"-Dspark.hadoop.hbase.regionserver.minorcompaction.pagecache.drop=true"
"-Dspark.hadoop.hbase.server.keyvalue.maxsize=10485760"
"-Dspark.history.fs.update.interval=10s"
"-Dspark.hadoop.hbase.rest.threads.max=100"
"-Dspark.hadoop.hbase.systemtables.compacting.memstore.type=NONE"
"-Dspark.hadoop.hbase.client.keyvalue.maxsize=10485760"
"-Dspark.hadoop.hbase.client.max.total.tasks=100"
"-Dspark.hadoop.hbase.defaults.for.version.skip=false"
"-Dspark.hadoop.hbase.zookeeper.property.initLimit=10"
"-Dspark.hadoop.hbase.mob.compaction.batch.size=100"
"-Dspark.hadoop.hbase.master.info.bindAddress=0.0.0.0"
"-Dspark.hadoop.hbase.regionserver.handler.count=30"
"-Dspark.hadoop.hbase.regionserver.region.split.policy=org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy"
"-Dspark.hadoop.hbase.rpc.shortoperation.timeout=10000"
"-Dspark.hadoop.hbase.zookeeper.property.clientPort=2181"
"-Dspark.hadoop.hbase.master.infoserver.redirect=true"
"-Dspark.hadoop.hbase.hstore.compaction.min.size=134217728"
"-Dspark.hadoop.hbase.hstore.compaction.max=10"
"-Dspark.hadoop.hbase.thrift.maxQueuedRequests=1000"
"-Dspark.driver.supervise=false" "-Dspark.sql.catalogImplementation=hive"
"-Dspark.hadoop.hbase.local.dir=/tmp/hbase-cdsuser/local/"
"-Dspark.hadoop.hbase.ipc.client.fallback-to-simple-auth-allowed=false"
"-Dspark.hadoop.hbase.hstore.compaction.kv.max=10"
"-Dspark.hadoop.hbase.regionserver.info.bindAddress=0.0.0.0"
"-Dspark.hadoop.hbase.hregion.memstore.block.multiplier=4"
"-Dspark.hadoop.hbase.regionserver.regionSplitLimit=1000"
"-Dspark.hadoop.hbase.bucketcache.combinedcache.enabled=true"
"-Dspark.eventLog.dir=/tmp"
"-Dspark.hadoop.hbase.regionserver.dns.interface=default"
"-Dspark.hadoop.hbase.storescanner.parallel.seek.threads=10"
"-Dspark.hadoop.hbase.rest.port=8080"
"-Dspark.hadoop.hbase.http.staticuser.user=dr.stack"
"-Dspark.hadoop.hbase.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*"
"-Dspark.client.authentication.client_id=9a1af6d4-59d9-49fc-b8ab-40469737cc9c"
"-Dspark.hadoop.hbase.ipc.server.fallback-to-simple-auth-allowed=false"
"-Dspark.hadoop.hbase.regionserver.hlog.writer.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter"
"-Dspark.driver.memory=4g"
"-Dspark.hadoop.hbase.client.perserver.requests.threshold=2147483647"
"-Dspark.hadoop.hbase.regionserver.dns.nameserver=default"
"-Dspark.hadoop.hbase.client.scanner.max.result.size=2097152"
"-Dspark.hadoop.hbase.auth.key.update.interval=86400000"
"-Dspark.hadoop.hbase.balancer.period=300000" "-
Dspark.hadoop.hbase.snapshot.restore.failsafe.name=hbase-failsafe-{
snapshot.name}-{restore.timestamp}"
"-Dspark.hadoop.hbase.status.publisher.class=org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher"
"-Dspark.hadoop.hbase.rest.support.proxyuser=false"
"-Dspark.hadoop.hbase.regionserver.thrift.framed=false"
"-Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider"
"-Dspark.hadoop.hbase.status.multicast.address.port=16100"
"-Dspark.hadoop.hbase.master.hostname=nl-itc-l-cdma00002.labs.liberty.biz"
"-Dspark.rpc.askTimeout=10s"
"-Dspark.hadoop.hbase.client.max.perserver.tasks=2"
"-Dspark.hadoop.hbase.serial.replication.waitingMs=10000"
"-Dspark.hadoop.hbase.mob.compactor.class=org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor"
"-Dspark.history.ui.port=18080"
"-Dspark.hadoop.hbase.zookeeper.property.maxClientCnxns=300"
"-Dspark.hadoop.hbase.regions.slop=0.001"
"-Dspark.hadoop.hbase.tmp.dir=/tmp/hbase-cdsuser"
"-Dspark.hadoop.hbase.offpeak.end.hour=-1"
"-Dspark.hadoop.zookeeper.znode.acl.parent=acl"
"-Dspark.hadoop.hbase.server.thread.wakefrequency=10000"
"-Dspark.hadoop.hbase.hregion.majorcompaction.jitter=0.50"
"-Dspark.hadoop.hbase.region.replica.replication.enabled=false"
"-Dspark.hadoop.hbase.zookeeper.leaderport=3888"
"-Dspark.hadoop.hbase.regionserver.thread.compaction.throttle=2684354560"
"-Dspark.hadoop.hbase.offpeak.start.hour=-1"
"-Dspark.hadoop.hbase.rest.threads.min=2"
"-Dspark.hadoop.hbase.regionserver.info.port=60030"
"-Dspark.hadoop.hbase.lease.recovery.timeout=900000"
"-Dspark.executor.cores=6" "-Dspark.hadoop.hbase.rootdir.perms=700"
"-Dspark.hadoop.hbase.status.listener.class=org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener"
"-Dspark.hadoop.hbase.hstore.time.to.purge.deletes=0"
"-Dspark.hadoop.hbase.zookeeper.dns.nameserver=default"
"-Dspark.hadoop.hbase.table.max.rowsize=1073741824"
"-Dspark.hadoop.hbase.hstore.compaction.min=3"
"-Dspark.hadoop.hbase.dfs.client.read.shortcircuit.buffer.size=131072"
"-Dspark.hadoop.hbase.cells.scanned.per.heartbeat.check=10000"
"-Dspark.hadoop.zookeeper.znode.parent=/hbase"
"-Dspark.hadoop.hbase.status.multicast.address.ip=226.1.1.3"
"org.apache.spark.deploy.worker.DriverWrapper" "spark://
Worker@172.30.61.226:41266"
"/opt/CDS/spark/work/driver-20220527123634-0000/hive-exec-3.1.1.jar"
"org.apache.hive.spark.client.RemoteDriver" "--remote-host" "
nl-itc-l-cdma00001.labs.liberty.biz" "--remote-port" "36667" "--conf"
"hive.spark.client.connect.timeout=30000" "--conf"
"hive.spark.client.server.connect.timeout=300000" "--conf"
"hive.spark.client.channel.log.level=null" "--conf"
"hive.spark.client.rpc.max.size=52428800" "--conf"
"hive.spark.client.rpc.threads=8" "--conf"
"hive.spark.client.secret.bits=256" "--conf"
"hive.spark.client.rpc.server.address=172.30.61.226"
========================================
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/CDS/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/CDS/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
at
org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:48)
at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:138)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
... 6 more
and eventually timesout
following is the content of *spark-defaults.conf*
*######################################################################################################################*
spark.master spark://172.30.61.226:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://cdscluster/spark/
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory hdfs://cdscluster/spark-logs
spark.history.fs.update.interval 10s
spark.history.ui.port 18080
spark.sql.catalogImplementation=hive
spark.sql.warehouse.dir /hive/cdswarehouse
spark.serializer org.apache.spark.serializer.KryoSerializer
*######################################################################################################################*
*hive-site.xml*
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/hive/cdswarehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby://
172.30.61.100:1527/cds_hive_metastoredb;create=true</value>
</property>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.ClientDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<property>
<name>spark.master</name>
<value>spark://172.30.61.226:7077</value>
</property>
<property>
<name>hive.metastore.port</name>
<value>9083</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.input.dir.recursive</name>
<value>true</value>
</property>
<property>
<name>spark.eventLog.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.spark.client.rpc.server.address</name>
<value>172.30.61.226</value>
</property>
<property>
<name>spark.eventLog.dir</name>
<value>/tmp</value>
</property>
<property>
<name>spark.executor.memory</name>
<value>6048m</value>
</property>
<property>
<name>spark.executor.cores</name>
<value>6</value>
</property>
<property>
<name>hive.spark.client.connect.timeout</name>
<value>30000ms</value>
</property>
<property>
<name>hive.spark.client.server.connect.timeout</name>
<value>300000ms</value>
</property>
</configuration>
*######################################################################################################################*
Have followed the instructions in [
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started]
and
have linked the jars
lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
spark-core_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-core_2.11-2.4.0.jar
lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
scala-library-2.11.12.jar -> /opt/CDS/spark/jars/scala-library-2.11.12.jar
lrwxrwxrwx 1 cdsuser cdsuser 55 May 27 12:35
spark-network-common_2.11-2.4.0.jar ->
/opt/CDS/spark/jars/spark-network-common_2.11-2.4.0.jar
lrwxrwxrwx 1 cdsuser cdsuser 47 May 27 12:35
spark-unsafe_2.11-2.4.0.jar ->
/opt/CDS/spark/jars/spark-unsafe_2.11-2.4.0.jar
*######################################################################################################################*
*Hive console*
*##################################*
hive> select count(*) from cds.spectrum_data;
2022-05-27 12:36:31,955 INFO [e32ede4e-2110-4aea-9068-40e4a4986f78 main]
reducesink.VectorReduceSinkEmptyKeyOperator:
VectorReduceSinkEmptyKeyOperator constructor vectorReduceSinkInfo
org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@62dfe152
Query ID = cdsuser_20220527123629_e7e2aa6a-0223-4943-b387-a24f09e26aa8
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Failed to execute spark task, with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark
client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)'
FAILED: Execution Error, return code 30041 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark
client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c
############################################################################################################
Would really appreciate it if you can guide me on this.
--
Live every day as if it were your last, because one of these days, it will
be.
Regards,
Prasanth M Sasidharan
