Anish Shrigondekar created SPARK-38809: ------------------------------------------
Summary: Implement option to skip null values in symmetric hash impl of stream-stream joins Key: SPARK-38809 URL: https://issues.apache.org/jira/browse/SPARK-38809 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.2.1 Reporter: Anish Shrigondekar Implement option to skip null values in symmetric hash impl of stream-stream joins * In the symmetric has join state manager, we can receive entries with null values for a key and that caused the `removeByValue` and get iterators to fail and run into the NullPointerException. * This is possible if the state recovered is written from a old spark version or its corrupted on disk. Since we don't have a utility to query this state, we would like to provide a conf option to skip nulls for the symmetric hash impl in stream stream joins. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org