I am pretty certain you need to change the write.format from “es” to “org.elasticsearch.spark.sql”Sent from my iPhoneOn 8 Sep 2023, at 03:10, Dipayan Dev wrote:++ DevOn Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev wrote:Hi, Can you please elaborate your last response? I don’
I’m pretty sure snappy file is not splittable. That’s why you have a single
task (and most likely core) reading the 1.9GB snappy file
Sent from my iPhone
> On 23 Mar 2023, at 07:36, yangjie01 wrote:
>
> Is there only one RowGroup for this file? You can check this by printing the
> file's met
Hi All,
First time contributing, so reaching out by email before creating a JIRA ticket
and PR. I would like to propose a small change/enhancement to
OptimizeSkewedJoin.
Currently, OptimizeSkewedJoin has a hardcoded limit for multi table joins
(limit = 2). For processes that have multiple join