Re: Error - Spark STREAMING

2022-09-21 Thread Anupam Singh 
Which version of spark are you using?

On Tue, Sep 20, 2022, 1:57 PM Akash Vellukai 
wrote:

> Hello,
>
>
>   py4j.protocol.Py4JJavaError: An error occurred while calling o80.load. :
> java.lang.NoClassDefFoundError:
> org/apache/spark/sql/internal/connector/SimpleTableProvider
>
>
> May anyone help Me to solve this issue.
>
>
> Thanks and regards
> Akash
>

Re: Dynamic shuffle partitions in a single job

2022-09-10 Thread Anupam Singh 
Commenting for better reach :)

On Thu, Sep 8, 2022, 11:56 AM Vibhor Gupta 
wrote:

> Hi Community,
>
> Is it possible to set no of shuffle partitions per exchange ?
>
> My spark query contains a lot of joins/aggregations involving big tables
> and small tables, so keeping a high value of spark.sql.shuffle.partitions
> helps with big tables, for small tables it creates a lot of overhead on the
> scheduler.
>
> Cheers,
> Vibhor
>
>