Let me try that and get back. Just wondering, if there a change in the way we pass the format in connector from Spark 2 to 3?
On Fri, 8 Sep 2023 at 12:35 PM, Alfie Davidson <alfie.davids...@gmail.com> wrote: > I am pretty certain you need to change the write.format from “es” to > “org.elasticsearch.spark.sql” > > Sent from my iPhone > > On 8 Sep 2023, at 03:10, Dipayan Dev <dev.dipaya...@gmail.com> wrote: > > > > ++ Dev > > On Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev <dev.dipaya...@gmail.com> > wrote: > >> Hi, >> >> Can you please elaborate your last response? I don’t have any external >> dependencies added, and just updated the Spark version as mentioned below. >> >> Can someone help me with this? >> >> On Fri, 1 Sep 2023 at 5:58 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> could the provided scope be the issue? >>> >>> On Sun, Aug 27, 2023 at 2:58 PM Dipayan Dev <dev.dipaya...@gmail.com> >>> wrote: >>> >>>> Using the following dependency for Spark 3 in POM file (My Scala >>>> version is 2.12.14) >>>> >>>> >>>> >>>> >>>> >>>> >>>> *<dependency> <groupId>org.elasticsearch</groupId> >>>> <artifactId>elasticsearch-spark-30_2.12</artifactId> >>>> <version>7.12.0</version> <scope>provided</scope></dependency>* >>>> >>>> >>>> The code throws error at this line : >>>> df.write.format("es").mode("overwrite").options(elasticOptions).save("index_name") >>>> The same code is working with Spark 2.4.0 and the following dependency >>>> >>>> >>>> >>>> >>>> >>>> *<dependency> <groupId>org.elasticsearch</groupId> >>>> <artifactId>elasticsearch-spark-20_2.12</artifactId> >>>> <version>7.12.0</version></dependency>* >>>> >>>> >>>> On Mon, 28 Aug 2023 at 12:17 AM, Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> What’s the version of the ES connector you are using? >>>>> >>>>> On Sat, Aug 26, 2023 at 10:17 AM Dipayan Dev <dev.dipaya...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> We're using Spark 2.4.x to write dataframe into the Elasticsearch >>>>>> index. >>>>>> As we're upgrading to Spark 3.3.0, it throwing out error >>>>>> Caused by: java.lang.ClassNotFoundException: es.DefaultSource >>>>>> at >>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) >>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) >>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) >>>>>> >>>>>> Looking at a few responses from Stackoverflow >>>>>> <https://stackoverflow.com/a/66452149>. it seems this is not yet >>>>>> supported by Elasticsearch-hadoop. >>>>>> >>>>>> Does anyone have experience with this? Or faced/resolved this issue >>>>>> in Spark 3? >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> Regards >>>>>> Dipayan >>>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> >>>> >>> CONFIDENTIALITY NOTICE: This electronic communication and any files >>> transmitted with it are confidential, privileged and intended solely for >>> the use of the individual or entity to whom they are addressed. If you are >>> not the intended recipient, you are hereby notified that any disclosure, >>> copying, distribution (electronic or otherwise) or forwarding of, or the >>> taking of any action in reliance on the contents of this transmission is >>> strictly prohibited. Please notify the sender immediately by e-mail if you >>> have received this email by mistake and delete this email from your system. >>> >>> Is it necessary to print this email? If you care about the environment >>> like we do, please refrain from printing emails. It helps to keep the >>> environment forested and litter-free. >> >>