[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368823#comment-16368823 ]
Kazuaki Ishizaki edited comment on SPARK-23427 at 2/19/18 7:01 AM: ------------------------------------------------------------------- I got the OOM with the same stack trace for both configurations when I ran this program using 256gb heap. Of course, since we know that to throw OOM is a problem, I will look at this. We would like to confirm whether this issue depends on the option or not. was (Author: kiszk): I got the OOM with the same stack trace for both configurations when I ran this program using 256gb heap. > spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver > ------------------------------------------------------------------------- > > Key: SPARK-23427 > URL: https://issues.apache.org/jira/browse/SPARK-23427 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Environment: SPARK 2.0 version > Reporter: Dhiraj > Priority: Critical > > We are facing issue around value of spark.sql.autoBroadcastJoinThreshold. > With spark.sql.autoBroadcastJoinThreshold -1 ( disable) we seeing driver > memory used flat. > With any other values 10MB, 5MB, 2 MB, 1MB, 10K, 1K we see driver memory used > goes up with rate depending upon the size of the autoBroadcastThreshold and > getting OOM exception. The problem is memory used by autoBroadcast is not > being free up in the driver. > Application imports oracle tables as master dataframes which are persisted. > Each job applies filter to these tables and then registered them as > tempViewTable . Then sql query are using to process data further. At the end > all the intermediate dataFrame are unpersisted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org