Hi Sean,
Removed the provided thing, but still the same issue.
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-30_${scala.compat.version}</artifactId>
<version>7.12.1</version>
</dependency>
On Fri, Sep 8, 2023 at 4:41 AM Sean Owen <sro...@gmail.com> wrote:
> By marking it provided, you are not including this dependency with your
> app. If it is also not somehow already provided by your spark cluster (this
> is what it means), then yeah this is not anywhere on the class path at
> runtime. Remove the provided scope.
>
> On Thu, Sep 7, 2023, 4:09 PM Dipayan Dev <dev.dipaya...@gmail.com> wrote:
>
>> Hi,
>>
>> Can you please elaborate your last response? I don’t have any external
>> dependencies added, and just updated the Spark version as mentioned below.
>>
>> Can someone help me with this?
>>
>> On Fri, 1 Sep 2023 at 5:58 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> could the provided scope be the issue?
>>>
>>> On Sun, Aug 27, 2023 at 2:58 PM Dipayan Dev <dev.dipaya...@gmail.com>
>>> wrote:
>>>
>>>> Using the following dependency for Spark 3 in POM file (My Scala
>>>> version is 2.12.14)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *<dependency> <groupId>org.elasticsearch</groupId>
>>>> <artifactId>elasticsearch-spark-30_2.12</artifactId>
>>>> <version>7.12.0</version> <scope>provided</scope></dependency>*
>>>>
>>>>
>>>> The code throws error at this line :
>>>> df.write.format("es").mode("overwrite").options(elasticOptions).save("index_name")
>>>> The same code is working with Spark 2.4.0 and the following dependency
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *<dependency> <groupId>org.elasticsearch</groupId>
>>>> <artifactId>elasticsearch-spark-20_2.12</artifactId>
>>>> <version>7.12.0</version></dependency>*
>>>>
>>>>
>>>> On Mon, 28 Aug 2023 at 12:17 AM, Holden Karau <hol...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> What’s the version of the ES connector you are using?
>>>>>
>>>>> On Sat, Aug 26, 2023 at 10:17 AM Dipayan Dev <dev.dipaya...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We're using Spark 2.4.x to write dataframe into the Elasticsearch
>>>>>> index.
>>>>>> As we're upgrading to Spark 3.3.0, it throwing out error
>>>>>> Caused by: java.lang.ClassNotFoundException: es.DefaultSource
>>>>>> at
>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>
>>>>>> Looking at a few responses from Stackoverflow
>>>>>> <https://stackoverflow.com/a/66452149>. it seems this is not yet
>>>>>> supported by Elasticsearch-hadoop.
>>>>>>
>>>>>> Does anyone have experience with this? Or faced/resolved this issue
>>>>>> in Spark 3?
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Regards
>>>>>> Dipayan
>>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>>
>>> CONFIDENTIALITY NOTICE: This electronic communication and any files
>>> transmitted with it are confidential, privileged and intended solely for
>>> the use of the individual or entity to whom they are addressed. If you are
>>> not the intended recipient, you are hereby notified that any disclosure,
>>> copying, distribution (electronic or otherwise) or forwarding of, or the
>>> taking of any action in reliance on the contents of this transmission is
>>> strictly prohibited. Please notify the sender immediately by e-mail if you
>>> have received this email by mistake and delete this email from your system.
>>>
>>> Is it necessary to print this email? If you care about the environment
>>> like we do, please refrain from printing emails. It helps to keep the
>>> environment forested and litter-free.
>>
>>
