Thanks Takiar, So, do you suggest I stick with Spark 1.6 or Spark 2.0.0? Which Hive version is the most appropriate?
Stéphane From: Sahil Takiar [mailto:takiar.sa...@gmail.com] Sent: Monday, November 27, 2017 18:20 To: user@hive.apache.org Subject: Re: Can't have Hive running with Spark Right now we only support Spark 2.0.0, the issue you are facing is probably due to a version mismatch issue. You may find this JIRA useful - SPARK-16292 - you want to make sure you are building the Spark distribution correctly. On Mon, Nov 27, 2017 at 8:48 AM, <stephane.d...@orange.com<mailto:stephane.d...@orange.com>> wrote: Hello all I’m trying to have Hive running on top of a Spark cluster. - Hive version: 2.3.2, installed with the embedded derby database (local mode) - Spark version: 2.2.0, installed in cluster mode, no yarn, no mesos - Hadoop version: 2.7.4 - OS: Redhat 7 There is something special here, I don’t run it on the top of Hadoop, but on the top of Elasticsearch thanks to the Elastic-hadoop bridge. The reason why I’m using derby and basic cluster mode for Spark is that I’m currently in a kind of discovery phase. What is working nicely: - Spark on ES: I can submit python scripts to query my Elastic db - Hive on ES: it works with engine=mr, I’d like to have it with engine=spark What I can see is that when I launch my Hive query, it seems first normal from the Hiveserver point of view: 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: { 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: "action" : "CreateSubmissionResponse", 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: "message" : "Driver successfully submitted as driver-20171127164308-0002", 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: "serverSparkVersion" : "2.2.0", 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: "submissionId" : "driver-20171127164308-0002", 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: "success" : true 2017-11-27T16:43:08,808 INFO [stderr-redir-1] client.SparkClientImpl: } But actually, on Spark side, I get the following error : Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS I’ve set the hive.spark.client.rpc.server.address property on all the Spark nodes, where I’ve installed also the Hive binaries and pushed the hive-site.xml. I’ve set the HIVE_CONF_DIR and the HIVE_HOME on all nodes also, but it doesn’t work. I’m a little bit lost now, I don’t see what I could do else ☹ Your help is appreciated. Thanks a lot, Stéphane _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -- Sahil Takiar Software Engineer takiar.sa...@gmail.com<mailto:takiar.sa...@gmail.com> | (510) 673-0309 _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
