Jungtaek Lim created SPARK-56361:
------------------------------------
Summary: Onboard error class for NPE in stream-stream join, with
context
Key: SPARK-56361
URL: https://issues.apache.org/jira/browse/SPARK-56361
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Jungtaek Lim
When there is some issue on stream-stream join (most likely an out of sync
between two state stores), stream-stream join can throw NPE.
Throwing NPE does not help anything for users and devs. For users, NPE is
mostly cryptic error and they know nothing what to do to mitigate the issue.
There is no information around context, so debugging is almost impossible for
devs when this happens.
We have captured several places which could throw NPE - we should provide the
better error for users which is less cryptic and gives a quick mitigation if
they are willing to tolerate losing some data to keep the query running. For
devs, we should leave the context to the log so that when users come to devs
with the error, devs can at least know where to start with.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
