Jean-Denis Giguère created SEDONA-79:
----------------------------------------
Summary: Pure SQL does not work with Thrift Server
Key: SEDONA-79
URL: https://issues.apache.org/jira/browse/SEDONA-79
Project: Apache Sedona
Issue Type: Bug
Reporter: Jean-Denis Giguère
h2. Current situation
When I add Sedona SQL support to the Thrift Server on a local Spark Cluster, a
{{scala.ScalaReflectionException}} associated with the class
{{org.locationtech.jts.geom.Geometry}} is triggered.
{code:bash}
/opt/spark/bin/spark-submit --class
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--master $SPARK_MASTER \
--hiveconf hive.server2.thrift.port=10001 --hiveconf
hive.server2.thrift.bind.host=0.0.0.0 \
--packages
org.apache.sedona:sedona-python-adapter-3.0_2.12:1.1.1-incubating,org.apache.sedona:sedona-viz-3.0_2.12:1.1.1-incubating,org.datasyslab:geotools-wrapper:geotools-24.0
\
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf
spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
\
--conf
spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
{code}
The error is not triggered when I launch the server, but when I attempt to
connect to it with the Apache Hive JDBC driver.
This error is not present when I use {{spark-shell}} with the same parameter.
h2. Expected situation
I would expect to be able to start the thrift server to expose sedona SQL on a
standalone cluster.
h2. Logs
{noformat}
docker-compose up spark-thrift
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a
swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
spark-dbt_spark-master_1 is up-to-date
spark-dbt_spark-worker-b_1 is up-to-date
spark-dbt_spark-worker-a_1 is up-to-date
Starting spark-dbt_spark-thrift_1 ... done
Attaching to spark-dbt_spark-thrift_1
spark-thrift_1 | WARNING: An illegal reflective access operation has occurred
spark-thrift_1 | WARNING: Illegal reflective access by
org.apache.spark.unsafe.Platform
(file:/opt/spark/jars/spark-unsafe_2.12-3.0.2.jar) to constructor
java.nio.DirectByteBuffer(long,int)
spark-thrift_1 | WARNING: Please consider reporting this to the maintainers
of org.apache.spark.unsafe.Platform
spark-thrift_1 | WARNING: Use --illegal-access=warn to enable warnings of
further illegal reflective access operations
spark-thrift_1 | WARNING: All illegal access operations will be denied in a
future release
spark-thrift_1 | Ivy Default Cache set to: /root/.ivy2/cache
spark-thrift_1 | The jars for the packages stored in: /root/.ivy2/jars
spark-thrift_1 | :: loading settings :: url =
jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
spark-thrift_1 | org.apache.sedona#sedona-python-adapter-3.0_2.12 added as a
dependency
spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12 added as a dependency
spark-thrift_1 | org.datasyslab#geotools-wrapper added as a dependency
spark-thrift_1 | :: resolving dependencies ::
org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0;1.0
spark-thrift_1 | confs: [default]
spark-thrift_1 | found
org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating in central
spark-thrift_1 | found org.locationtech.jts#jts-core;1.18.0 in central
spark-thrift_1 | found org.wololo#jts2geojson;0.16.1 in central
spark-thrift_1 | found
com.fasterxml.jackson.core#jackson-databind;2.12.2 in central
spark-thrift_1 | found
com.fasterxml.jackson.core#jackson-annotations;2.12.2 in central
spark-thrift_1 | found com.fasterxml.jackson.core#jackson-core;2.12.2 in
central
spark-thrift_1 | found
org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating in central
spark-thrift_1 | found
org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 in central
spark-thrift_1 | found
org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating in central
spark-thrift_1 | found
org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating in central
spark-thrift_1 | found org.beryx#awt-color-factory;1.0.0 in central
spark-thrift_1 | found org.datasyslab#geotools-wrapper;geotools-24.0 in
central
spark-thrift_1 | :: resolution report :: resolve 533ms :: artifacts dl 15ms
spark-thrift_1 | :: modules in use:
spark-thrift_1 | com.fasterxml.jackson.core#jackson-annotations;2.12.2
from central in [default]
spark-thrift_1 | com.fasterxml.jackson.core#jackson-core;2.12.2 from
central in [default]
spark-thrift_1 | com.fasterxml.jackson.core#jackson-databind;2.12.2 from
central in [default]
spark-thrift_1 | org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating
from central in [default]
spark-thrift_1 |
org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating from central
in [default]
spark-thrift_1 | org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating
from central in [default]
spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating
from central in [default]
spark-thrift_1 | org.beryx#awt-color-factory;1.0.0 from central in
[default]
spark-thrift_1 | org.datasyslab#geotools-wrapper;geotools-24.0 from
central in [default]
spark-thrift_1 | org.locationtech.jts#jts-core;1.18.0 from central in
[default]
spark-thrift_1 |
org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 from central in
[default]
spark-thrift_1 | org.wololo#jts2geojson;0.16.1 from central in [default]
spark-thrift_1 | :: evicted modules:
spark-thrift_1 | org.locationtech.jts#jts-core;1.18.1 by
[org.locationtech.jts#jts-core;1.18.0] in [default]
spark-thrift_1 |
---------------------------------------------------------------------
spark-thrift_1 | | | modules ||
artifacts |
spark-thrift_1 | | conf | number| search|dwnlded|evicted||
number|dwnlded|
spark-thrift_1 |
---------------------------------------------------------------------
spark-thrift_1 | | default | 13 | 0 | 0 | 1 ||
12 | 0 |
spark-thrift_1 |
---------------------------------------------------------------------
spark-thrift_1 | :: retrieving ::
org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0
spark-thrift_1 | confs: [default]
spark-thrift_1 | 0 artifacts copied, 12 already retrieved (0kB/13ms)
spark-thrift_1 | 22/01/24 20:55:06 WARN NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
spark-thrift_1 | Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Started daemon
with process name: 7@9d3fb159061a
spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
handler for TERM
spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
handler for HUP
spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
handler for INT
spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Starting
SparkContext
spark-thrift_1 | 22/01/24 20:55:06 INFO HiveConf: Found configuration file
null
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Running Spark version
3.0.2
spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils:
==============================================================
spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils: Resources for
spark.driver:
spark-thrift_1 |
spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils:
==============================================================
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Submitted application:
SparkSQL::172.28.0.5
spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view acls
to: root
spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify
acls to: root
spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view acls
groups to:
spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify
acls groups to:
spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(root); groups with view permissions: Set(); users with modify permissions:
Set(root); groups with modify permissions: Set()
spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started service
'sparkDriver' on port 34759.
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
MapOutputTracker
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
BlockManagerMaster
spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
BlockManagerMasterHeartbeat
spark-thrift_1 | 22/01/24 20:55:07 INFO DiskBlockManager: Created local
directory at /tmp/blockmgr-2f049248-da56-420b-b1ec-2c5fe1e924fc
spark-thrift_1 | 22/01/24 20:55:07 INFO MemoryStore: MemoryStore started
with capacity 434.4 MiB
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
OutputCommitCoordinator
spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started service
'SparkUI' on port 4040.
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkUI: Bound SparkUI to
spark-thrift, and started at http://9d3fb159061a:4040
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
at
spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
at
spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar at
spark://9d3fb159061a:34759/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.locationtech.jts_jts-core-1.18.0.jar at
spark://9d3fb159061a:34759/jars/org.locationtech.jts_jts-core-1.18.0.jar with
timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.wololo_jts2geojson-0.16.1.jar at
spark://9d3fb159061a:34759/jars/org.wololo_jts2geojson-0.16.1.jar with
timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
at
spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
at
spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
at
spark://9d3fb159061a:34759/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
at
spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
at
spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar at
spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar
with timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
file:///root/.ivy2/jars/org.beryx_awt-color-factory-1.0.0.jar at
spark://9d3fb159061a:34759/jars/org.beryx_awt-color-factory-1.0.0.jar with
timestamp 1643057707087
spark-thrift_1 | 22/01/24 20:55:07 WARN SparkContext: Please ensure that the
number of slots available on your executors is limited by the number of cores
to task cpus and not another custom resource. If cores is not the limiting
resource then dynamic allocation will not work properly!
spark-thrift_1 | 22/01/24 20:55:07 INFO StandaloneAppClient$ClientEndpoint:
Connecting to master spark://spark-master:7077...
spark-thrift_1 | 22/01/24 20:55:07 INFO TransportClientFactory: Successfully
created connection to spark-master/172.28.0.2:7077 after 24 ms (0 ms spent in
bootstraps)
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
Connected to Spark cluster with app ID app-20220124205508-0004
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint:
Executor added: app-20220124205508-0004/0 on
worker-20220124201503-172.28.0.3-7000 (172.28.0.3:7000) with 2 core(s)
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: Granted
executor ID app-20220124205508-0004/0 on hostPort 172.28.0.3:7000 with 2
core(s), 1024.0 MiB RAM
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint:
Executor added: app-20220124205508-0004/1 on
worker-20220124201513-172.28.0.4-7000 (172.28.0.4:7000) with 2 core(s)
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: Granted
executor ID app-20220124205508-0004/1 on hostPort 172.28.0.4:7000 with 2
core(s), 1024.0 MiB RAM
spark-thrift_1 | 22/01/24 20:55:08 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 39909.
spark-thrift_1 | 22/01/24 20:55:08 INFO NettyBlockTransferService: Server
created on 9d3fb159061a:39909
spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registering
BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMasterEndpoint:
Registering block manager 9d3fb159061a:39909 with 434.4 MiB RAM,
BlockManagerId(driver, 9d3fb159061a, 39909, None)
spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registered
BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Initialized
BlockManager: BlockManagerId(driver, 9d3fb159061a, 39909, None)
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint:
Executor updated: app-20220124205508-0004/1 is now RUNNING
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint:
Executor updated: app-20220124205508-0004/0 is now RUNNING
spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
SchedulerBackend is ready for scheduling beginning after reached
minRegisteredResourcesRatio: 0.0
spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Setting
hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir
('file:/opt/spark/spark-warehouse').
spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Warehouse path is
'file:/opt/spark/spark-warehouse'.
spark-thrift_1 | 22/01/24 20:55:08 INFO HiveUtils: Initializing
HiveMetastoreConnection version 2.3.7 using Spark classes.
spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS
directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created local
directory: /tmp/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS
directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2/_tmp_space.db
spark-thrift_1 | 22/01/24 20:55:09 INFO HiveClientImpl: Warehouse location
for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name
hive.stats.jdbc.timeout does not exist
spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name
hive.stats.retries.wait does not exist
spark-thrift_1 | 22/01/24 20:55:10 INFO HiveMetaStore: 0: Opening raw store
with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
spark-thrift_1 | 22/01/24 20:55:10 INFO ObjectStore: ObjectStore, initialize
called
spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property
datanucleus.cache.level2 unknown - will be ignored
spark-thrift_1 | 22/01/24 20:55:10 INFO ResourceProfile: Default
ResourceProfile created, executor resources: Map(cores -> name: cores, amount:
1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor:
), task resources: Map(cpus -> name: cpus, amount: 1.0)
spark-thrift_1 | 22/01/24 20:55:11 INFO
CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
NettyRpcEndpointRef(spark-client://Executor) (172.28.0.4:46986) with ID 1
spark-thrift_1 | 22/01/24 20:55:11 INFO
CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
NettyRpcEndpointRef(spark-client://Executor) (172.28.0.3:48170) with ID 0
spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint:
Registering block manager 172.28.0.4:34799 with 434.4 MiB RAM,
BlockManagerId(1, 172.28.0.4, 34799, None)
spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint:
Registering block manager 172.28.0.3:34961 with 434.4 MiB RAM,
BlockManagerId(0, 172.28.0.3, 34961, None)
spark-thrift_1 | 22/01/24 20:55:12 INFO ObjectStore: Setting MetaStore
object pin classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct
SQL, underlying DB is DERBY
spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized ObjectStore
spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore: Version information not
found in metastore. hive.metastore.schema.verification is not enabled so
recording the schema version 2.3.0
spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore:
setMetaStoreSchemaVersion called but recording version is disabled: version =
2.3.0, comment = Set by MetaStore [email protected]
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added admin role in
metastore
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added public role in
metastore
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: No user is added in
admin role, since config is empty
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_all_functions
spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
ip=unknown-ip-addr cmd=get_all_functions
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_database:
default
spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
ip=unknown-ip-addr cmd=get_database: default
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveUtils: Initializing execution
hive, version 2.3.7
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveClientImpl: Warehouse location
for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: Operation log root
directory is created: /tmp/root/operation_logs
spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
Background operation thread pool size: 100
spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
Background operation thread wait queue size: 100
spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
Background operation thread keepalive time: 10 seconds
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:OperationManager is inited.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:SessionManager is inited.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service: CLIService
is inited.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:ThriftBinaryCLIService is inited.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:
HiveServer2 is inited.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:OperationManager is started.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:SessionManager is started.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:CLIService
is started.
spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: ObjectStore, initialize
called
spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct
SQL, underlying DB is DERBY
spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized ObjectStore
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_databases:
default
spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
ip=unknown-ip-addr cmd=get_databases: default
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Cleaning up thread
local RawStore...
spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Done cleaning up
thread local RawStore
spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
Service:ThriftBinaryCLIService is started.
spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:HiveServer2
is started.
spark-thrift_1 | 22/01/24 20:55:14 INFO HiveThriftServer2: HiveThriftServer2
started
spark-thrift_1 | 22/01/24 20:55:14 INFO ThriftCLIService: Starting
ThriftBinaryCLIService on port 10001 with 5...500 worker threads
spark-thrift_1 | 22/01/24 20:56:24 INFO ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V8
spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS
directory: /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created local
directory: /tmp/root/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS
directory:
/tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2/_tmp_space.db
spark-thrift_1 | 22/01/24 20:56:24 INFO HiveSessionImpl: Operation log
session directory is created:
/tmp/root/operation_logs/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
spark-thrift_1 | Exception in thread "HiveServer2-Handler-Pool: Thread-87"
java.lang.ExceptionInInitializerError
spark-thrift_1 | at
org.apache.sedona.sql.UDF.UdfRegistrator$.registerAll(UdfRegistrator.scala:32)
spark-thrift_1 | at
org.apache.sedona.sql.utils.SedonaSQLRegistrator$.registerAll(SedonaSQLRegistrator.scala:34)
spark-thrift_1 | at
org.apache.sedona.sql.SedonaSqlExtensions.$anonfun$apply$1(SedonaSqlExtensions.scala:28)
spark-thrift_1 | at
org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildCheckRules$1(SparkSessionExtensions.scala:174)
spark-thrift_1 | at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
spark-thrift_1 | at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
spark-thrift_1 | at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
spark-thrift_1 | at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
spark-thrift_1 | at
scala.collection.TraversableLike.map(TraversableLike.scala:238)
spark-thrift_1 | at
scala.collection.TraversableLike.map$(TraversableLike.scala:231)
spark-thrift_1 | at
scala.collection.AbstractTraversable.map(Traversable.scala:108)
spark-thrift_1 | at
org.apache.spark.sql.SparkSessionExtensions.buildCheckRules(SparkSessionExtensions.scala:174)
spark-thrift_1 | at
org.apache.spark.sql.internal.BaseSessionStateBuilder.customCheckRules(BaseSessionStateBuilder.scala:227)
spark-thrift_1 | at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:100)
spark-thrift_1 | at
org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:74)
spark-thrift_1 | at
org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:336)
spark-thrift_1 | at
org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:83)
spark-thrift_1 | at
org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:83)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
spark-thrift_1 | at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:138)
spark-thrift_1 | at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:138)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
spark-thrift_1 | at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
spark-thrift_1 | at
org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
spark-thrift_1 | at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
spark-thrift_1 | at
org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
spark-thrift_1 | at
org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
spark-thrift_1 | at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
spark-thrift_1 | at
org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
spark-thrift_1 | at
org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
spark-thrift_1 | at
org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
spark-thrift_1 | at
org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
spark-thrift_1 | at
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:353)
spark-thrift_1 | at
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:247)
spark-thrift_1 | at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
spark-thrift_1 | at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
spark-thrift_1 | at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
spark-thrift_1 | at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
spark-thrift_1 | at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
spark-thrift_1 | at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
spark-thrift_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
spark-thrift_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
spark-thrift_1 | at java.base/java.lang.Thread.run(Unknown Source)
spark-thrift_1 | Caused by: scala.ScalaReflectionException: class
org.locationtech.jts.geom.Geometry in JavaMirror with
org.apache.hadoop.hive.ql.exec.UDFClassLoader@5a71c867 of type class
org.apache.hadoop.hive.ql.exec.UDFClassLoader with classpath [] and parent
being jdk.internal.loader.ClassLoaders$AppClassLoader@3c153a1 of type class
jdk.internal.loader.ClassLoaders$AppClassLoader with classpath [<unknown>] and
parent being jdk.internal.loader.ClassLoaders$PlatformClassLoader@435dacd9 of
type class jdk.internal.loader.ClassLoaders$PlatformClassLoader with classpath
[<unknown>] and parent being primordial classloader with boot classpath
[<unknown>] not found.
spark-thrift_1 | at
scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:146)
spark-thrift_1 | at
scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:29)
spark-thrift_1 | at
org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec$$typecreator1$1.apply(AggregateFunctions.scala:42)
spark-thrift_1 | at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
spark-thrift_1 | at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
spark-thrift_1 | at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
spark-thrift_1 | at
org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec.$init$(AggregateFunctions.scala:42)
spark-thrift_1 | at
org.apache.spark.sql.sedona_sql.expressions.ST_Union_Aggr.<init>(AggregateFunctions.scala:56)
spark-thrift_1 | at
org.apache.sedona.sql.UDF.Catalog$.<init>(Catalog.scala:126)
spark-thrift_1 | at
org.apache.sedona.sql.UDF.Catalog$.<clinit>(Catalog.scala)
spark-thrift_1 | ... 46 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)