spark.sql.hive.exec.dynamic.partition description

Hi Guys, Does any one have detailed descriptions for hive parameters in spark? like spark.sql.hive.exec.dynamic.partition I couldn't find any reference in my spark 2.3.2 configuration. I'm looking into a problem that Spark cannot understand Hive partition at all. In my Hive table it is partitione

Fwd: autoBroadcastJoinThreshold not working as expected

Dear all, I'm on a case that when certain table being exposed to broadcast join, the query will eventually failed with remote block error. Firstly. We set the spark.sql.autoBroadcastJoinThreshold to 10MB, namely 10485760 [image: image.png] Then we proceed to perform query. In the SQL plan, we fo

Fwd: autoBroadcastJoinThreshold not working as expected

Dear all, I'm on a case that when certain table being exposed to broadcast join, the query will eventually failed with remote block error. Firstly. We set the spark.sql.autoBroadcastJoinThreshold to 10MB, namely 10485760 [image: image.png] Then we proceed to perform query. In the SQL plan, we fo

autoBroadcastJoinThreshold not working as expected

Dear all, I'm on a case that when certain table being exposed to broadcast join, the query will eventually failed with remote block error. Firstly. We set the spark.sql.autoBroadcastJoinThreshold to 10MB, namely 10485760 [image: image.png] Then we proceed to perform query. In the SQL plan, we fo