Sometimes TaskContext configuration is almost empty

Hi, I'm using PySpark to construct a DataFrame and run it. This DataFrame uses views created by other DataFrames in the same session. One of these views, say view A, has an SQL statement that has in one of the SQL expressions a combination of map functions, among them Map Concat. Another view, say

[Issue] Spark SQL - broadcast failure

Hi all, Been facing a weird issue lately. In our production code base , we have an explicit broadcast for a small table. It is just a look up table that is around 1gb in size in s3 and just had few million records and 5 columns. The ETL was running fine , but with no change from the codebase nor