Hi David,
Are you talking about the dynamic run-time filter pushdown of spark ? If that is the case, I had already raised a discussion fee months back to do the similarly in Carbondata to improve carbon join by using carbondata's metadata. But since spark is already doing it , it's decided not to do any changes in carbon side for the same feature and make it complex. So this design is decided in the community meeting last week, please correct me if I'm wrong in understanding your reply. Also it's not plan change, basically, it's just adding a where filter of custom UDF of block paths in the current join query that's all which skips the unwanted block files to scan. Regards, Akash On Thu, Feb 18, 2021, 6:36 AM David CaiQiang <david.c...@gmail.com> wrote: > Hi Akash, > You can enhance the runtime filter to improve the join performance. > > It has the rule to dynamically check whether the join can add the > runtime filter or not. > > Better to push down the runtime filter into CarbonDataSourceScan, and > better to avoid adding a UDF function to rewrite the plan. > > > > > > ----- > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
