Re: Error - Spark STREAMING

2022-09-21 Thread Anupam Singh
Which version of spark are you using? On Tue, Sep 20, 2022, 1:57 PM Akash Vellukai wrote: > Hello, > > > py4j.protocol.Py4JJavaError: An error occurred while calling o80.load. : > java.lang.NoClassDefFoundError: > org/apache/spark/sql/internal/connector/SimpleTableProvider > > > May anyone

Re: Dynamic shuffle partitions in a single job

2022-09-10 Thread Anupam Singh
Commenting for better reach :) On Thu, Sep 8, 2022, 11:56 AM Vibhor Gupta wrote: > Hi Community, > > Is it possible to set no of shuffle partitions per exchange ? > > My spark query contains a lot of joins/aggregations involving big tables > and small tables, so keeping a high value of