[Spark Core] Adaptive dynamic partition pruning

2022-11-09 Thread hajyoussef amine
Hello everyone, Let me take the following spark sql example to demonstrate the issue we're having: ``` Select * FROM small_table Inner join big_table on small_table.foreign_key = big_table.partition_key Inner join bigger_table on big_table.foreign_key = bigger_table.partition_key where

Re: [Spark Core] Adaptive dynamic partition pruning

2022-11-11 Thread hajyoussef amine
eries. So, your fact table is big_table which is used to join the others. > So there’s only one subqueryboradcast dynamicpruning plan before > big_table’s scan while there’s none for the others. > > I’m not sure that I’m correct. Hope it’s helpful to you. > > 2022年

Re: [Spark Core] Adaptive dynamic partition pruning

2022-11-11 Thread hajyoussef amine
c pruning works in queries with multi joins. > BTW, could you execute ‘explain extended your sql’? > > > 2022年11月10日 02:10,hajyoussef amine 写道: > > > > Hello everyone, > > > > Let me take the following spark sql example to demonstrate the issue > we