Commenting for better reach :) On Thu, Sep 8, 2022, 11:56 AM Vibhor Gupta <vibhor.gu...@walmart.com.invalid> wrote:
> Hi Community, > > Is it possible to set no of shuffle partitions per exchange ? > > My spark query contains a lot of joins/aggregations involving big tables > and small tables, so keeping a high value of spark.sql.shuffle.partitions > helps with big tables, for small tables it creates a lot of overhead on the > scheduler. > > Cheers, > Vibhor > >
