[ https://issues.apache.org/jira/browse/SPARK-28094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941623#comment-16941623 ]
Jungtaek Lim edited comment on SPARK-28094 at 10/1/19 8:22 AM: --------------------------------------------------------------- FYI: After SPARK-28074, Spark will log warn message for query which has the pattern of "possibly" emitting late rows. SPARK-24634 proposes to expose the number of late rows (older than watermark + allowed delay) in both SQL execution detail page and streaming query listener, which helps identifying the correctness issue. was (Author: kabhwan): FYI: After SPARK-28074, Spark will log warn message for query which has the pattern of "possibly" emitting late rows. SPARK-24634 will also expose the number of late rows (older than watermark + allowed delay) in both SQL execution detail page and streaming query listener. > Multiple left joins or aggregations in one query produce incorrect results > -------------------------------------------------------------------------- > > Key: SPARK-28094 > URL: https://issues.apache.org/jira/browse/SPARK-28094 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.4.3 > Reporter: Joe Ammann > Priority: Major > > Structured streaming queries which involve more than one non-map-like > transformations (either left joins or aggregations) produce incorrect (and > sometimes inconsistent) results. > I have put up a sample here as Gist > https://gist.github.com/jammann/b58bfbe0f4374b89ecea63c1e32c8f17 > The test case involves 4 topics, and I've implement 6 different use cases of > which 3 fail to produce the expected results > 1) A innerjoin B: to topic A_B > expected: 10 inner joined messages > observation: OK > 2) A innerjoin B outerjoin C: to topic A_B_outer_C > expected: 9 inner joined messages, last one in watermark > observation: OK > 3) A innerjoin B outerjoin C outerjoin D: to topic A_B_outer_C_outer_D > expected: 9 inner/outer joined messages, 3 match C, 1 match D, last one > in watermark > observation: NOK, only 3 messages are produced, where the C outer > join matches > 4) B outerjoin C: to topic B_outer_C > expected: 9 outer joined messages, 3 match C, 1 match D, last one in > watermark > observation: NOK, same as 3 > 5) A innerjoin B agg on field of A: to topic A_inner_B_agg > expected: 3 groups with a total of 9 inner joined messages, last one in > watermark > observation: OK > 6) A innerjoin B outerjoin C agg on field of A: to topic A_inner_B_outer_C_agg > expected: 3 groups with a total of 9 inner joined messages, last one in > watermark > observation: NOK, 3 groups with 1 message each, where C outer join > matches -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org