[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135428#comment-17135428 ]
Yuanjian Li commented on SPARK-31990: ------------------------------------- [~maropu] Thanks for the quick fix! [~dongjoon] This issue is found when investigating SPARK-31894, and it's also a blocker issue for it. I added a detailed analysis in SPARK-31894's [PR comments|https://github.com/apache/spark/pull/28707#issuecomment-643916110]. I think the simple revert fix in [https://github.com/apache/spark/pull/28830] can be merged without tests. I'll add a new integrated test in SPARK-31905. > Streaming's state store compatibility is broken > ----------------------------------------------- > > Key: SPARK-31990 > URL: https://issues.apache.org/jira/browse/SPARK-31990 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.0.0 > Reporter: Xiao Li > Priority: Blocker > Labels: correctness > > [This > line|https://github.com/apache/spark/pull/28062/files#diff-7a46f10c3cedbf013cf255564d9483cdR2458] > of [https://github.com/apache/spark/pull/28062] changed the order of > groupCols in dropDuplicates(). Thus, the executor JVM could probably crash, > throw a random exception or even return a wrong answer when using the > checkpoint written by the previous version. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org