Joe Ammann created SPARK-28094:
----------------------------------

             Summary: Multiple left joins or aggregations in one query produce 
incorrect results
                 Key: SPARK-28094
                 URL: https://issues.apache.org/jira/browse/SPARK-28094
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.4.3
            Reporter: Joe Ammann


Structured streaming queries which involve more than one non-map-like 
transformations (either left joins or aggregations) produce incorrect (and 
sometimes inconsistent) results.

I have put up a sample here as Gist 
https://gist.github.com/jammann/b58bfbe0f4374b89ecea63c1e32c8f17

The test case involves 4 topics, and I've implement 6 different use cases of 
which 3 fail to produce the expected results

1) A innerjoin B: to topic A_B
        expected: 10 inner joined messages
        observation: OK

2) A innerjoin B outerjoin C: to topic A_B_outer_C 
        expected: 9 inner joined messages, last one in watermark
        observation: OK

3) A innerjoin B outerjoin C outerjoin D: to topic A_B_outer_C_outer_D
        expected: 9 inner/outer joined messages, 3 match C, 1 match D, last one 
in watermark
        observation: NOK, only 3 messages are produced, where the C outer join 
matches

4) B outerjoin C: to topic B_outer_C 
        expected: 9 outer joined messages, 3 match C, 1 match D, last one in 
watermark
        observation: NOK, same as 3

5) A innerjoin B agg on field of A: to topic A_inner_B_agg
        expected: 3 groups with a total of 9 inner joined messages, last one in 
watermark
        observation: OK

6) A innerjoin B outerjoin C agg on field of A: to topic A_inner_B_outer_C_agg
        expected: 3 groups with a total of 9 inner joined messages, last one in 
watermark
        observation: NOK, 3 groups with 1 message each, where C outer join 
matches




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to