Hello all
I'm trying to have Hive running on top of a Spark cluster.
- Hive version: 2.3.2, installed with the embedded derby database
(local mode)
- Spark version: 2.2.0, installed in cluster mode, no yarn, no mesos
- Hadoop version: 2.7.4
- OS: Redhat 7
There is something special here, I don't run it on the top of Hadoop, but on
the top of Elasticsearch thanks to the Elastic-hadoop bridge. The reason why
I'm using derby and basic cluster mode for Spark is that I'm currently in a
kind of discovery phase.
What is working nicely:
- Spark on ES: I can submit python scripts to query my Elastic db
- Hive on ES: it works with engine=mr, I'd like to have it with
engine=spark
What I can see is that when I launch my Hive query, it seems first normal from
the Hiveserver point of view:
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: {
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl:
"action" : "CreateSubmissionResponse",
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl:
"message" : "Driver successfully submitted as driver-20171127164308-0002",
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl:
"serverSparkVersion" : "2.2.0",
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl:
"submissionId" : "driver-20171127164308-0002",
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl:
"success" : true
2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: }
But actually, on Spark side, I get the following error :
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
I've set the hive.spark.client.rpc.server.address property on all the Spark
nodes, where I've installed also the Hive binaries and pushed the
hive-site.xml. I've set the HIVE_CONF_DIR and the HIVE_HOME on all nodes also,
but it doesn't work.
I'm a little bit lost now, I don't see what I could do else :(
Your help is appreciated.
Thanks a lot,
Stéphane
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
