Anil K created SPARK-62032:
------------------------------
Summary: Operators that pair data with an initial state by key do
not check that the two sides' grouping keys have one type
Key: SPARK-62032
URL: https://issues.apache.org/jira/browse/SPARK-62032
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Anil K
Operators that pair a grouped dataset with an initial state by key do not check
that the two sides' grouping keys have one type: flatMapGroupsWithState and
mapGroupsWithState with an initialState, transformWithState with an
initialState (Scala, classic and Spark Connect), and transformWithStateInPandas
/ transformWithStateInPySpark with an initialState (Python). With
{{groupBy(cols).as[K, V]}}, or a Python {{groupBy}}, the data side and the
initial-state side can group by keys of different types, e.g. an {{INT}} and a
{{BIGINT}}.
Each side is then hash-partitioned on its own key type, and state is paired
with data by comparing raw key bytes, so state can be missed or attached to the
wrong key. Measured on master (7c8056a3365) for a batch flatMapGroupsWithState
with an initial state, over the same key pairs as SPARK-60782:
* {{BIGINT}} data keys with {{INT}} state keys: 5 of 7 keys found their state;
* {{INT}}/{{BIGINT}}, {{INT}}/{{DOUBLE}}, struct and array keys: with AQE on, a
task fails with {{AssertionError}} at {{CoGroupedIterator.next}};
* {{STRING}}/{{INT}}: {{NegativeArraySizeException}};
* decimals of another storage or scale, {{DATE}}/{{TIMESTAMP}},
{{TIMESTAMP}}/{{TIMESTAMP_NTZ}} outside UTC: some keys miss their state.
SPARK-60782 fixes the same gap for cogroups, by widening Pandas and Arrow
cogroup keys where SQL has a common type, and otherwise failing at analysis
with {{COGROUP_KEY_TYPE_MISMATCH}}. These operators are stateful, so the fix is
likely different: widening the data side's key would change the state store's
key schema, and rejecting can fail a streaming query restarted from a
checkpoint. A fix could reuse SPARK-60782's check, rejecting keys not equal
ignoring nullability and field names, possibly widening only the initial-state
side to the data side's type.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]