Re: TaskMemoryManager: Failed to allocate a page

Usually using broadcast join could boost the performance when you have enough memory, You should decrease it or even disable it when there is no enough memory. On Thu, Oct 27, 2016 at 1:22 PM, Pietro Pugni wrote: > Thank you Davies, > this worked! But what are the

Re: TaskMemoryManager: Failed to allocate a page

Thank you Davies, this worked! But what are the consequences of setting spark.sql.autoBroadcastJoinThreshold=0? Will it degrade or boost performance? Thank you again Pietro > Il giorno 27 ott 2016, alle ore 18:54, Davies Liu ha > scritto: > > I think this is caused by

Re: TaskMemoryManager: Failed to allocate a page

I think this is caused by BroadcastHashJoin try to use more memory than the amount driver have, could you decrease the spark.sql.autoBroadcastJoinThreshold (-1 or 0 means disable it)? On Thu, Oct 27, 2016 at 9:19 AM, Pietro Pugni wrote: > I’m sorry, here’s the formatted

Re: TaskMemoryManager: Failed to allocate a page

I’m sorry, here’s the formatted message text: I'm running an ETL process that joins table1 with other tables (CSV files), one table at time (for example table1 with table2, table1 with table3, and so on). The join is written inside a PostgreSQL istance using JDBC. The entire process runs