Hi Prasanth, I would suggest not to invest too heavily in Hive on Spark. I recent years there was no movement around the feature and it will be removed in Hive 4.0.0.
Thanks, Peter > On 2022. May 27., at 13:00, Prasanth M Sasidharan <prasha...@gmail.com> wrote: > > Hello team, > > I am trying to use spark as the engine for Hive. > Hive version : 3.1.1 > Spark: 2.4.0 (have tried with several versions) > > I am able to query the hive tables from within pyspark.However, when i > execute the hive query from the hive shell, it throws the following error > > Launch Command: "/opt/CDS/pre_requisites/jdk1.8.0_141/bin/java" "-cp" > "/opt/CDS/spark/conf/:/opt/CDS/spark/jars/*:/opt/CDS/hadoop-2.9.2/etc/hadoop/:/etc/tez/conf:/opt/CDS/hive/conf/:/opt/CDS/hive/lib/HikariCP-2.6.1.jar:/opt/CDS/hive/lib/ST4-4.0.4.jar:/opt/CDS/hive/lib/accumulo-core-1.7.3.jar:/opt/CDS/hive/lib/accumulo-fate-1.7.3.jar:/opt/CDS/hive/lib/accumulo-start-1.7.3.jar:/opt/CDS/hive/lib/accumulo-trace-1.7.3.jar:/opt/CDS/hive/lib/aircompressor-0.10.jar:/opt/CDS/hive/lib/ant-1.9.1.jar:/opt/CDS/hive/lib/ant-launcher-1.9.1.jar:/opt/CDS/hive/lib/antlr-runtime-3.5.2.jar:/opt/CDS/hive/lib/antlr4-runtime-4.5.jar:/opt/CDS/hive/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/CDS/hive/lib/apache-jsp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/apache-jstl-9.3.20.v20170531.jar:/opt/CDS/hive/lib/arrow-format-0.8.0.jar:/opt/CDS/hive/lib/arrow-memory-0.8.0.jar:/opt/CDS/hive/lib/arrow-vector-0.8.0.jar:/opt/CDS/hive/lib/asm-5.0.1.jar:/opt/CDS/hive/lib/asm-commons-5.0.1.jar:/opt/CDS/hive/lib/asm-tree-5.0.1.jar:/opt/CDS/hive/lib/audience-annotations-0.5.0.jar:/opt/CDS/hive/lib/avatica-1.11.0.jar:/opt/CDS/hive/lib/avro-1.7.7.jar:/opt/CDS/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/CDS/hive/lib/calcite-core-1.16.0.jar:/opt/CDS/hive/lib/calcite-druid-1.16.0.jar:/opt/CDS/hive/lib/calcite-linq4j-1.16.0.jar:/opt/CDS/hive/lib/commons-cli-1.2.jar:/opt/CDS/hive/lib/commons-codec-1.7.jar:/opt/CDS/hive/lib/commons-collections4-4.1.jar:/opt/CDS/hive/lib/commons-compiler-2.7.6.jar:/opt/CDS/hive/lib/commons-compress-1.9.jar:/opt/CDS/hive/lib/commons-crypto-1.0.0.jar:/opt/CDS/hive/lib/commons-dbcp-1.4.jar:/opt/CDS/hive/lib/commons-io-2.4.jar:/opt/CDS/hive/lib/commons-lang-2.6.jar:/opt/CDS/hive/lib/commons-lang3-3.2.jar:/opt/CDS/hive/lib/commons-logging-1.0.4.jar:/opt/CDS/hive/lib/commons-math-2.1.jar:/opt/CDS/hive/lib/commons-math3-3.6.1.jar:/opt/CDS/hive/lib/commons-pool-1.5.4.jar:/opt/CDS/hive/lib/commons-vfs2-2.1.jar:/opt/CDS/hive/lib/curator-client-2.12.0.jar:/opt/CDS/hive/lib/curator-framework-2.12.0.jar:/opt/CDS/hive/lib/curator-recipes-2.12.0.jar:/opt/CDS/hive/lib/datanucleus-api-jdo-4.2.4.jar:/opt/CDS/hive/lib/datanucleus-core-4.1.17.jar:/opt/CDS/hive/lib/datanucleus-rdbms-4.1.19.jar:/opt/CDS/hive/lib/derby-10.14.1.0.jar:/opt/CDS/hive/lib/disruptor-3.3.6.jar:/opt/CDS/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/CDS/hive/lib/druid-hdfs-storage-0.12.0.jar:/opt/CDS/hive/lib/ecj-4.4.2.jar:/opt/CDS/hive/lib/esri-geometry-api-2.0.0.jar:/opt/CDS/hive/lib/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hive/lib/flatbuffers-1.2.0-3f79e055.jar:/opt/CDS/hive/lib/groovy-all-2.4.11.jar:/opt/CDS/hive/lib/gson-2.2.4.jar:/opt/CDS/hive/lib/guava-19.0.jar:/opt/CDS/hive/lib/hbase-client-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-http-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-mapreduce-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-api-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-prefix-tree-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-procedure-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-shaded-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-replication-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-server-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-shaded-miscellaneous-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-netty-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-protobuf-1.0.1.jar:/opt/CDS/hive/lib/hive-accumulo-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-beeline-3.1.1.jar:/opt/CDS/hive/lib/hive-classification-3.1.1.jar:/opt/CDS/hive/lib/hive-cli-3.1.1.jar:/opt/CDS/hive/lib/hive-common-3.1.1.jar:/opt/CDS/hive/lib/hive-contrib-3.1.1.jar:/opt/CDS/hive/lib/hive-druid-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-exec-3.1.1.jar:/opt/CDS/hive/lib/hive-hbase-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-core-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-server-extensions-3.1.1.jar:/opt/CDS/hive/lib/hive-hplsql-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-kryo-registrator-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1-tests.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-ext-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-server-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-tez-3.1.1.jar:/opt/CDS/hive/lib/hive-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-serde-3.1.1.jar:/opt/CDS/hive/lib/hive-service-3.1.1.jar:/opt/CDS/hive/lib/hive-service-rpc-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-0.23-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-common-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-scheduler-3.1.1.jar:/opt/CDS/hive/lib/hive-standalone-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-storage-api-2.7.0.jar:/opt/CDS/hive/lib/hive-streaming-3.1.1.jar:/opt/CDS/hive/lib/hive-testutils-3.1.1.jar:/opt/CDS/hive/lib/hive-upgrade-acid-3.1.1.jar:/opt/CDS/hive/lib/hive-vector-code-gen-3.1.1.jar:/opt/CDS/hive/lib/hk2-api-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-locator-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-utils-2.5.0-b32.jar:/opt/CDS/hive/lib/hppc-0.7.2.jar:/opt/CDS/hive/lib/htrace-core-3.2.0-incubating.jar:/opt/CDS/hive/lib/httpclient-4.5.2.jar:/opt/CDS/hive/lib/httpcore-4.4.4.jar:/opt/CDS/hive/lib/ivy-2.4.0.jar:/opt/CDS/hive/lib/jackson-annotations-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-asl-1.9.13.jar:/opt/CDS/hive/lib/jackson-databind-2.9.5.jar:/opt/CDS/hive/lib/jackson-dataformat-smile-2.9.5.jar:/opt/CDS/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/CDS/hive/lib/jamon-runtime-2.3.1.jar:/opt/CDS/hive/lib/janino-2.7.6.jar:/opt/CDS/hive/lib/javassist-3.20.0-GA.jar:/opt/CDS/hive/lib/javax.annotation-api-1.2.jar:/opt/CDS/hive/lib/javax.inject-2.5.0-b32.jar:/opt/CDS/hive/lib/javax.jdo-3.2.0-m3.jar:/opt/CDS/hive/lib/javax.servlet-api-3.1.0.jar:/opt/CDS/hive/lib/javax.servlet.jsp-2.3.2.jar:/opt/CDS/hive/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/CDS/hive/lib/javax.ws.rs-api-2.0.1.jar:/opt/CDS/hive/lib/javolution-5.5.1.jar:/opt/CDS/hive/lib/jcodings-1.0.18.jar:/opt/CDS/hive/lib/jcommander-1.32.jar:/opt/CDS/hive/lib/jdo-api-3.0.1.jar:/opt/CDS/hive/lib/jersey-client-2.25.1.jar:/opt/CDS/hive/lib/jersey-common-2.25.1.jar:/opt/CDS/hive/lib/jersey-container-servlet-core-2.25.1.jar:/opt/CDS/hive/lib/jersey-guava-2.25.1.jar:/opt/CDS/hive/lib/jersey-media-jaxb-2.25.1.jar:/opt/CDS/hive/lib/jersey-server-2.25.1.jar:/opt/CDS/hive/lib/jettison-1.1.jar:/opt/CDS/hive/lib/jetty-annotations-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-http-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-io-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jaas-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jndi-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-plus-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-rewrite-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-runner-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-schemas-3.1.jar:/opt/CDS/hive/lib/jetty-security-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-util-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-webapp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-xml-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jline-2.12.jar:/opt/CDS/hive/lib/joda-time-2.9.9.jar:/opt/CDS/hive/lib/joni-2.1.11.jar:/opt/CDS/hive/lib/jpam-1.1.jar:/opt/CDS/hive/lib/json-1.8.jar:/opt/CDS/hive/lib/jsr305-3.0.0.jar:/opt/CDS/hive/lib/jta-1.1.jar:/opt/CDS/hive/lib/kryo-shaded-4.0.2.jar:/opt/CDS/hive/lib/libfb303-0.9.3.jar:/opt/CDS/hive/lib/libthrift-0.9.3.jar:/opt/CDS/hive/lib/memory-0.9.0.jar:/opt/CDS/hive/lib/metrics-core-3.1.0.jar:/opt/CDS/hive/lib/metrics-json-3.1.0.jar:/opt/CDS/hive/lib/metrics-jvm-3.1.0.jar:/opt/CDS/hive/lib/minlog-1.3.0.jar:/opt/CDS/hive/lib/mysql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/netty-3.10.5.Final.jar:/opt/CDS/hive/lib/netty-all-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-buffer-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-common-4.1.17.Final.jar:/opt/CDS/hive/lib/opencsv-2.3.jar:/opt/CDS/hive/lib/orc-core-1.5.1.jar:/opt/CDS/hive/lib/orc-shims-1.5.1.jar:/opt/CDS/hive/lib/org.abego.treelayout.core-1.0.1.jar:/opt/CDS/hive/lib/osgi-resource-locator-1.0.1.jar:/opt/CDS/hive/lib/paranamer-2.3.jar:/opt/CDS/hive/lib/parquet-hadoop-bundle-1.10.0.jar:/opt/CDS/hive/lib/postgresql-9.4.1208.jre7.jar:/opt/CDS/hive/lib/postgresql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/protobuf-java-2.5.0.jar:/opt/CDS/hive/lib/scala-library-2.11.12.jar:/opt/CDS/hive/lib/sketches-core-0.9.0.jar:/opt/CDS/hive/lib/snappy-java-1.1.4.jar:/opt/CDS/hive/lib/spark-core_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-network-common_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-unsafe_2.11-2.4.0.jar:/opt/CDS/hive/lib/sqlline-1.3.0.jar:/opt/CDS/hive/lib/stax-api-1.0.1.jar:/opt/CDS/hive/lib/super-csv-2.2.0.jar:/opt/CDS/hive/lib/taglibs-standard-impl-1.2.5.jar:/opt/CDS/hive/lib/taglibs-standard-spec-1.2.5.jar:/opt/CDS/hive/lib/tempus-fugit-1.1.jar:/opt/CDS/hive/lib/transaction-api-1.1.jar:/opt/CDS/hive/lib/validation-api-1.1.0.Final.jar:/opt/CDS/hive/lib/velocity-1.5.jar:/opt/CDS/hive/lib/websocket-api-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-common-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/zookeeper-3.4.6.jar:/opt/CDS/hadoop-2.9.2/share/hadoop/tools/lib/hadoop-distcp-2.9.2.jar:/opt/CDS/hbase//conf/:/opt/CDS/hbase//lib/shaded-clients/hbase-shaded-mapreduce-2.1.1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/commons-logging-1.2.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/CDS/hive/lib/log4j-1.2-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-core-2.10.0.jar:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar:/opt/CDS/hive/lib/log4j-web-2.10.0.jar:/opt/CDS/hadoop-2.9.2/contrib/capacity-scheduler/*.jar:/opt/CDS/hadoop-2.9.2//share/hadoop/common/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/common/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/lib/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/*" > "-Xmx4096M" > "-Dspark.hadoop.hbase.http.filter.initializers=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter" > "-Dspark.hadoop.hbase.defaults.for.version=2.0.0-alpha4" > "-Dspark.hadoop.hbase.wal.dir.perms=700" > "-Dspark.hadoop.hbase.hstore.checksum.algorithm=CRC32C" > "-Dspark.hadoop.hbase.regionserver.thrift.framed.max_frame_size_in_mb=2" > "-Dspark.hadoop.hbase.snapshot.restore.take.failsafe.snapshot=true" > "-Dspark.hadoop.hbase.regionserver.port=60020" > "-Dspark.hadoop.hbase.rpc.rows.warning.threshold=5000" > "-Dspark.hadoop.hbase.storescanner.parallel.seek.enable=false" > "-Dspark.hadoop.hbase.client.max.perregion.tasks=1" > "-Dspark.hadoop.hbase.mob.compaction.threads.max=1" > "-Dspark.hadoop.hbase.mob.compaction.mergeable.threshold=1342177280" > "-Dspark.hadoop.hbase.rpc.timeout=60000" > "-Dspark.hadoop.hbase.replication.rpc.codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags" > "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" > "-Dspark.hadoop.hbase.auth.token.max.lifetime=604800000" > "-Dspark.hadoop.hbase.mob.compaction.chore.period=604800" > "-Dspark.hadoop.hbase.bulkload.retries.number=10" > "-Dspark.hadoop.hbase.hregion.memstore.flush.size=134217728" > "-Dspark.master=spark://172.30.61.226:7077 <http://172.30.61.226:7077/>" > "-Dspark.hadoop.hbase.master.port=60000" > "-Dspark.hadoop.hbase.server.versionfile.writeattempts=3" > "-Dspark.hadoop.hbase.regionserver.logroll.errors.tolerated=2" > "-Dspark.hadoop.hbase.unsafe.stream.capability.enforce=false" > "-Dspark.hadoop.hbase.master.info.port=60010" "-Dspark.app.name > <http://dspark.app.name/>=Hive on Spark (sessionId = > 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)" > "-Dspark.hadoop.hbase.table.lock.enable=true" > "-Dspark.hadoop.hbase.lease.recovery.dfs.timeout=64000" > "-Dspark.hadoop.hbase.regionserver.thrift.compact=false" > "-Dspark.hadoop.hbase.hregion.majorcompaction=604800000" > "-Dspark.hadoop.hbase.zookeeper.peerport=2888" > "-Dspark.hadoop.hbase.cluster.distributed=true" > "-Dspark.hadoop.hbase.regionserver.storefile.refresh.period=0" > "-Dspark.kryo.classesToRegister=org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,org.apache.hadoop.io.Writable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.io.HiveKey" > "-Dspark.hadoop.hbase.regionserver.optionalcacheflushinterval=3600000" > "-Dspark.hadoop.hbase.master.procedurewalcleaner.ttl=604800000" > "-Dspark.hadoop.hbase.zookeeper.quorum=172.30.61.229,172.30.61.230,172.30.61.231" > "-Dspark.sql.warehouse.dir=/hive/cdswarehouse" > "-Dspark.hadoop.fs.defaultFS=hdfs://cdscluster/" > "-Dspark.hadoop.hbase.hstore.blockingStoreFiles=10" > "-Dspark.hadoop.hbase.regionserver.hlog.reader.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader" > "-Dspark.hadoop.hbase.regionserver.checksum.verify=true" > "-Dspark.hadoop.hbase.rest.readonly=false" > "-Dspark.hadoop.hbase.regionserver.majorcompaction.pagecache.drop=true" > "-Dspark.submit.deployMode=cluster" > "-Dspark.hadoop.hbase.http.max.threads=16" > "-Dspark.hadoop.hbase.data.umask.enable=false" > "-Dspark.hadoop.hbase.hstore.flusher.count=2" > "-Dspark.hadoop.hbase.server.compactchecker.interval.multiplier=1000" > "-Dspark.hadoop.hbase.hstore.compaction.max.size=9223372036854775807" > "-Dspark.hadoop.hbase.rest.filter.classes=org.apache.hadoop.hbase.rest.filter.GzipFilter" > "-Dspark.hadoop.hbase.rest.csrf.enabled=false" > "-Dspark.hadoop.hbase.coordinated.state.manager.class=org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager" > "-Dspark.hadoop.hbase.mob.delfile.max.count=3" > "-Dspark.hadoop.hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer" > "-Dspark.hadoop.hbase.snapshot.region.timeout=300000" > "-Dspark.hadoop.hbase.hstore.compactionThreshold=3" > "-Dspark.hadoop.hbase.data.umask=000" > "-Dspark.client.authentication.secret=0ffffffea690fffffffb4d0ffffffc045240ffffffff410ffffffd56874181d04700ffffffae320ffffffc40ffffffe036c1a321c63e5e0ffffffde213a3d" > "-Dspark.hadoop.hbase.hstore.bytes.per.checksum=16384" > "-Dspark.hadoop.hbase.server.scanner.max.result.size=104857600" > "-Dspark.hadoop.hbase.ipc.server.callqueue.scan.ratio=0" > "-Dspark.hadoop.hbase.ipc.client.tcpnodelay=true" > "-Dspark.hadoop.hbase.hstore.compaction.ratio.offpeak=5.0F" > "-Dspark.hadoop.hbase.zookeeper.property.dataDir=/data/CDS/zookeeper" > "-Dspark.jars=file:/opt/CDS/hive/lib/hive-exec-3.1.1.jar" > "-Dspark.hadoop.hbase.client.pause=100" > "-Dspark.hadoop.hbase.coprocessor.enabled=true" > "-Dspark.hadoop.hbase.thrift.minWorkerThreads=16" > "-Dspark.kryo.referenceTracking=false" > "-Dspark.hadoop.hbase.master.mob.ttl.cleaner.period=86400" > "-Dspark.hadoop.hbase.status.published=false" > "-Dspark.hadoop.hbase.client.retries.number=35" > "-Dspark.hadoop.hbase.hstore.blockingWaitTime=90000" > "-Dspark.hadoop.hbase.mob.cache.evict.remain.ratio=0.5f" > "-Dspark.hadoop.hbase.master.normalizer.class=org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer" > "-Dspark.hadoop.hbase.client.scanner.caching=2147483647" > "-Dspark.hadoop.hbase.normalizer.period=300000" > "-Dspark.hadoop.hbase.hregion.max.filesize=10737418240" > "-Dspark.hadoop.hbase.hregion.memstore.mslab.enabled=true" > "-Dspark.hadoop.hbase.hstore.compaction.ratio=1.2F" > "-Dspark.hadoop.hbase.security.visibility.mutations.checkauths=false" > "-Dspark.hadoop.hbase.client.localityCheck.threadPoolSize=2" > "-Dspark.hadoop.hbase.master.hfilecleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner" > "-Dspark.hadoop.hbase.coprocessor.user.enabled=true" > "-Dspark.hadoop.hbase.rs.cacheblocksonwrite=false" > "-Dspark.executor.memory=6048m" > "-Dspark.hadoop.hbase.regionserver.msginterval=3000" > "-Dspark.hadoop.hbase.snapshot.master.timeout.millis=300000" > "-Dspark.executor.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/" > "-Dspark.hadoop.cloneConf=true" > "-Dspark.hadoop.hbase.fs.tmp.dir=/user/cdsuser/hbase-staging" > "-Dspark.hadoop.hbase.mob.cache.evict.period=3600" > "-Dspark.hadoop.hbase.security.exec.permission.checks=false" > "-Dspark.hadoop.hbase.hregion.percolumnfamilyflush.size.lower.bound.min=16777216" > "-Dspark.hadoop.hbase.zookeeper.property.syncLimit=5" > "-Dspark.history.fs.logDirectory=hdfs://cdscluster/spark-logs" > "-Dspark.hadoop.hbase.client.operation.timeout=1200000" > "-Dspark.hadoop.hbase.ipc.server.callqueue.handler.factor=0.1" > "-Dspark.hadoop.hbase.rootdir=hdfs://cdscluster/hbase" > "-Dspark.hadoop.hbase.ipc.server.callqueue.read.ratio=0" > "-Dspark.eventLog.enabled=true" > "-Dspark.hadoop.hbase.regionserver.handler.abort.on.error.percent=0.5" > "-Dspark.hadoop.hbase.thrift.maxWorkerThreads=1000" > "-Dspark.hadoop.hbase.regionserver.logroll.period=3600000" > "-Dspark.hadoop.hbase.mob.file.cache.size=1000" > "-Dspark.hadoop.hbase.master.logcleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner" > "-Dspark.hadoop.hbase.coprocessor.abortonerror=true" > "-Dspark.hadoop.hbase.hregion.preclose.flush.size=5242880" > "-Dspark.hadoop.hbase.dynamic.jars.dir=hdfs://cdscluster/hbase/lib" > "-Dspark.hadoop.hbase.column.max.version=1" > "-Dspark.hadoop.hbase.master.logcleaner.ttl=600000" > "-Dspark.hadoop.hbase.regionserver.hostname.disable.master.reversedns=false" > "-Dspark.hadoop.hbase.replication.source.maxthreads=10" > "-Dspark.hadoop.hbase.client.scanner.timeout.period=60000" > "-Dspark.hadoop.hbase.security.authentication=simple" > "-Dspark.hadoop.hbase.snapshot.enabled=true" > "-Dspark.hadoop.hbase.master.balancer.maxRitPercent=1.0" > "-Dspark.hadoop.hbase.zookeeper.dns.interface=default" > "-Dspark.driver.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/" > "-Dspark.hadoop.hbase.display.keys=true" > "-Dspark.hadoop.hbase.regionserver.info.port.auto=false" > "-Dspark.hadoop.hbase.client.write.buffer=2097152" > "-Dspark.hadoop.hbase.regionserver.minorcompaction.pagecache.drop=true" > "-Dspark.hadoop.hbase.server.keyvalue.maxsize=10485760" > "-Dspark.history.fs.update.interval=10s" > "-Dspark.hadoop.hbase.rest.threads.max=100" > "-Dspark.hadoop.hbase.systemtables.compacting.memstore.type=NONE" > "-Dspark.hadoop.hbase.client.keyvalue.maxsize=10485760" > "-Dspark.hadoop.hbase.client.max.total.tasks=100" > "-Dspark.hadoop.hbase.defaults.for.version.skip=false" > "-Dspark.hadoop.hbase.zookeeper.property.initLimit=10" > "-Dspark.hadoop.hbase.mob.compaction.batch.size=100" > "-Dspark.hadoop.hbase.master.info.bindAddress=0.0.0.0" > "-Dspark.hadoop.hbase.regionserver.handler.count=30" > "-Dspark.hadoop.hbase.regionserver.region.split.policy=org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy" > "-Dspark.hadoop.hbase.rpc.shortoperation.timeout=10000" > "-Dspark.hadoop.hbase.zookeeper.property.clientPort=2181" > "-Dspark.hadoop.hbase.master.infoserver.redirect=true" > "-Dspark.hadoop.hbase.hstore.compaction.min.size=134217728" > "-Dspark.hadoop.hbase.hstore.compaction.max=10" > "-Dspark.hadoop.hbase.thrift.maxQueuedRequests=1000" > "-Dspark.driver.supervise=false" "-Dspark.sql.catalogImplementation=hive" > "-Dspark.hadoop.hbase.local.dir=/tmp/hbase-cdsuser/local/" > "-Dspark.hadoop.hbase.ipc.client.fallback-to-simple-auth-allowed=false" > "-Dspark.hadoop.hbase.hstore.compaction.kv.max=10" > "-Dspark.hadoop.hbase.regionserver.info.bindAddress=0.0.0.0" > "-Dspark.hadoop.hbase.hregion.memstore.block.multiplier=4" > "-Dspark.hadoop.hbase.regionserver.regionSplitLimit=1000" > "-Dspark.hadoop.hbase.bucketcache.combinedcache.enabled=true" > "-Dspark.eventLog.dir=/tmp" > "-Dspark.hadoop.hbase.regionserver.dns.interface=default" > "-Dspark.hadoop.hbase.storescanner.parallel.seek.threads=10" > "-Dspark.hadoop.hbase.rest.port=8080" > "-Dspark.hadoop.hbase.http.staticuser.user=dr.stack" > "-Dspark.hadoop.hbase.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*" > "-Dspark.client.authentication.client_id=9a1af6d4-59d9-49fc-b8ab-40469737cc9c" > "-Dspark.hadoop.hbase.ipc.server.fallback-to-simple-auth-allowed=false" > "-Dspark.hadoop.hbase.regionserver.hlog.writer.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter" > "-Dspark.driver.memory=4g" > "-Dspark.hadoop.hbase.client.perserver.requests.threshold=2147483647" > "-Dspark.hadoop.hbase.regionserver.dns.nameserver=default" > "-Dspark.hadoop.hbase.client.scanner.max.result.size=2097152" > "-Dspark.hadoop.hbase.auth.key.update.interval=86400000" > "-Dspark.hadoop.hbase.balancer.period=300000" > "-Dspark.hadoop.hbase.snapshot.restore.failsafe.name > <http://dspark.hadoop.hbase.snapshot.restore.failsafe.name/>=hbase-failsafe-{snapshot.name > <http://snapshot.name/>}-{restore.timestamp}" > "-Dspark.hadoop.hbase.status.publisher.class=org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher" > "-Dspark.hadoop.hbase.rest.support.proxyuser=false" > "-Dspark.hadoop.hbase.regionserver.thrift.framed=false" > "-Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider" > "-Dspark.hadoop.hbase.status.multicast.address.port=16100" > "-Dspark.hadoop.hbase.master.hostname=nl-itc-l-cdma00002.labs.liberty.biz > <http://nl-itc-l-cdma00002.labs.liberty.biz/>" "-Dspark.rpc.askTimeout=10s" > "-Dspark.hadoop.hbase.client.max.perserver.tasks=2" > "-Dspark.hadoop.hbase.serial.replication.waitingMs=10000" > "-Dspark.hadoop.hbase.mob.compactor.class=org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor" > "-Dspark.history.ui.port=18080" > "-Dspark.hadoop.hbase.zookeeper.property.maxClientCnxns=300" > "-Dspark.hadoop.hbase.regions.slop=0.001" > "-Dspark.hadoop.hbase.tmp.dir=/tmp/hbase-cdsuser" > "-Dspark.hadoop.hbase.offpeak.end.hour=-1" > "-Dspark.hadoop.zookeeper.znode.acl.parent=acl" > "-Dspark.hadoop.hbase.server.thread.wakefrequency=10000" > "-Dspark.hadoop.hbase.hregion.majorcompaction.jitter=0.50" > "-Dspark.hadoop.hbase.region.replica.replication.enabled=false" > "-Dspark.hadoop.hbase.zookeeper.leaderport=3888" > "-Dspark.hadoop.hbase.regionserver.thread.compaction.throttle=2684354560" > "-Dspark.hadoop.hbase.offpeak.start.hour=-1" > "-Dspark.hadoop.hbase.rest.threads.min=2" > "-Dspark.hadoop.hbase.regionserver.info.port=60030" > "-Dspark.hadoop.hbase.lease.recovery.timeout=900000" > "-Dspark.executor.cores=6" "-Dspark.hadoop.hbase.rootdir.perms=700" > "-Dspark.hadoop.hbase.status.listener.class=org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener" > "-Dspark.hadoop.hbase.hstore.time.to.purge.deletes=0" > "-Dspark.hadoop.hbase.zookeeper.dns.nameserver=default" > "-Dspark.hadoop.hbase.table.max.rowsize=1073741824" > "-Dspark.hadoop.hbase.hstore.compaction.min=3" > "-Dspark.hadoop.hbase.dfs.client.read.shortcircuit.buffer.size=131072" > "-Dspark.hadoop.hbase.cells.scanned.per.heartbeat.check=10000" > "-Dspark.hadoop.zookeeper.znode.parent=/hbase" > "-Dspark.hadoop.hbase.status.multicast.address.ip=226.1.1.3" > "org.apache.spark.deploy.worker.DriverWrapper" > "spark://Worker@172.30.61.226:41266 <http://Worker@172.30.61.226:41266/>" > "/opt/CDS/spark/work/driver-20220527123634-0000/hive-exec-3.1.1.jar" > "org.apache.hive.spark.client.RemoteDriver" "--remote-host" > "nl-itc-l-cdma00001.labs.liberty.biz > <http://nl-itc-l-cdma00001.labs.liberty.biz/>" "--remote-port" "36667" > "--conf" "hive.spark.client.connect.timeout=30000" "--conf" > "hive.spark.client.server.connect.timeout=300000" "--conf" > "hive.spark.client.channel.log.level=null" "--conf" > "hive.spark.client.rpc.max.size=52428800" "--conf" > "hive.spark.client.rpc.threads=8" "--conf" > "hive.spark.client.secret.bits=256" "--conf" > "hive.spark.client.rpc.server.address=172.30.61.226" > ======================================== > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/CDS/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/CDS/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings > <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65) > at > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) > Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS > at > org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:48) > at > org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:138) > at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536) > ... 6 more > > and eventually timesout > > following is the content of spark-defaults.conf > ###################################################################################################################### > spark.master spark://172.30.61.226:7077 > <http://172.30.61.226:7077/> > spark.eventLog.enabled true > spark.eventLog.dir hdfs://cdscluster/spark/ > spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider > spark.history.fs.logDirectory hdfs://cdscluster/spark-logs > spark.history.fs.update.interval 10s > spark.history.ui.port 18080 > spark.sql.catalogImplementation=hive > spark.sql.warehouse.dir /hive/cdswarehouse > spark.serializer org.apache.spark.serializer.KryoSerializer > ###################################################################################################################### > hive-site.xml > <?xml version="1.0" encoding="UTF-8" standalone="no"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <configuration> > > <property> > <name>hive.metastore.warehouse.dir</name> > <value>/hive/cdswarehouse</value> > </property> > > <property> > <name>javax.jdo.option.ConnectionURL</name> > > <value>jdbc:derby://172.30.61.100:1527/cds_hive_metastoredb;create=true > <http://172.30.61.100:1527/cds_hive_metastoredb;create=true></value> > </property> > <property> > <name>datanucleus.schema.autoCreateTables</name> > <value>true</value> > </property> > > <property> > <name>javax.jdo.option.ConnectionDriverName</name> > <value>org.apache.derby.jdbc.ClientDriver</value> > <description>Driver class name for a JDBC metastore</description> > </property> > <property> > <name>hive.server2.thrift.port</name> > <value>10000</value> > </property> > > <property> > <name>hive.server2.enable.doAs</name> > <value>false</value> > </property> > > <property> > <name>hive.execution.engine</name> > <value>spark</value> > </property> > <property> > <name>spark.master</name> > <value>spark://172.30.61.226:7077 <http://172.30.61.226:7077/></value> > </property> > <property> > <name>hive.metastore.port</name> > <value>9083</value> > </property> > > <property> > <name>mapreduce.input.fileinputformat.input.dir.recursive</name> > <value>true</value> > </property> > <property> > <name>spark.eventLog.enabled</name> > <value>true</value> > </property> > <property> > <name>hive.spark.client.rpc.server.address</name> > <value>172.30.61.226</value> > </property> > > <property> > <name>spark.eventLog.dir</name> > <value>/tmp</value> > </property> > <property> > <name>spark.executor.memory</name> > <value>6048m</value> > </property> > <property> > <name>spark.executor.cores</name> > <value>6</value> > </property> > > <property> > <name>hive.spark.client.connect.timeout</name> > <value>30000ms</value> > </property> > <property> > <name>hive.spark.client.server.connect.timeout</name> > <value>300000ms</value> > </property> > </configuration> > ###################################################################################################################### > Have followed the instructions in > [https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started > > <https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started>] > and have linked the jars > > lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35 spark-core_2.11-2.4.0.jar > -> /opt/CDS/spark/jars/spark-core_2.11-2.4.0.jar > lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35 scala-library-2.11.12.jar > -> /opt/CDS/spark/jars/scala-library-2.11.12.jar > lrwxrwxrwx 1 cdsuser cdsuser 55 May 27 12:35 > spark-network-common_2.11-2.4.0.jar -> > /opt/CDS/spark/jars/spark-network-common_2.11-2.4.0.jar > lrwxrwxrwx 1 cdsuser cdsuser 47 May 27 12:35 > spark-unsafe_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-unsafe_2.11-2.4.0.jar > ###################################################################################################################### > > Hive console > ################################## > hive> select count(*) from cds.spectrum_data; > 2022-05-27 12:36:31,955 INFO [e32ede4e-2110-4aea-9068-40e4a4986f78 main] > reducesink.VectorReduceSinkEmptyKeyOperator: VectorReduceSinkEmptyKeyOperator > constructor vectorReduceSinkInfo > org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@62dfe152 > Query ID = cdsuser_20220527123629_e7e2aa6a-0223-4943-b387-a24f09e26aa8 > Total jobs = 1 > Launching Job 1 out of 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer=<number> > In order to limit the maximum number of reducers: > set hive.exec.reducers.max=<number> > In order to set a constant number of reducers: > set mapreduce.job.reduces=<number> > Failed to execute spark task, with exception > 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark > client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)' > FAILED: Execution Error, return code 30041 from > org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client > for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c > ############################################################################################################ > Would really appreciate it if you can guide me on this. > > > -- > Live every day as if it were your last, because one of these days, it will be. > > Regards, > > Prasanth M Sasidharan
