[ https://issues.apache.org/jira/browse/SPARK-24809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547337#comment-16547337 ]
zenglinxi commented on SPARK-24809: ----------------------------------- [^Spark LongHashedRelation serialization.svg] I think it's a hidden but critical bug that may cause data error. > Serializing LongHashedRelation in executor may result in data error > ------------------------------------------------------------------- > > Key: SPARK-24809 > URL: https://issues.apache.org/jira/browse/SPARK-24809 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 > Environment: Spark 2.2.1 > hadoop 2.7.1 > Reporter: Lijia Liu > Priority: Critical > Attachments: Spark LongHashedRelation serialization.svg > > > When join key is long or int in broadcast join, Spark will use > LongHashedRelation as the broadcast value. Details see SPARK-14419. But, if > the broadcast value is abnormal big, executor will serialize it to disk. But, > data will lost when serializing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org