[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518483#comment-17518483 ] Apache Spark commented on SPARK-38809: -- User 'anishshri-db' has created a pull request for this issue: https://github.com/apache/spark/pull/36090 > Implement option to skip null values in symmetric hash impl of stream-stream > joins > -- > > Key: SPARK-38809 > URL: https://issues.apache.org/jira/browse/SPARK-38809 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 3.2.1 >Reporter: Anish Shrigondekar >Priority: Major > > Implement option to skip null values in symmetric hash impl of stream-stream > joins > * In the symmetric has join state manager, we can receive entries with null > values for a key and that caused the `removeByValue` and get iterators to > fail and run into the NullPointerException. > * This is possible if the state recovered is written from a old spark > version or its corrupted on disk. Since we don't have a utility to query this > state, we would like to provide a conf option to skip nulls for the symmetric > hash impl in stream stream joins. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518481#comment-17518481 ] Apache Spark commented on SPARK-38809: -- User 'anishshri-db' has created a pull request for this issue: https://github.com/apache/spark/pull/36090 > Implement option to skip null values in symmetric hash impl of stream-stream > joins > -- > > Key: SPARK-38809 > URL: https://issues.apache.org/jira/browse/SPARK-38809 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 3.2.1 >Reporter: Anish Shrigondekar >Priority: Major > > Implement option to skip null values in symmetric hash impl of stream-stream > joins > * In the symmetric has join state manager, we can receive entries with null > values for a key and that caused the `removeByValue` and get iterators to > fail and run into the NullPointerException. > * This is possible if the state recovered is written from a old spark > version or its corrupted on disk. Since we don't have a utility to query this > state, we would like to provide a conf option to skip nulls for the symmetric > hash impl in stream stream joins. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518344#comment-17518344 ] Anish Shrigondekar commented on SPARK-38809: Working on this PR and will send the change out soon > Implement option to skip null values in symmetric hash impl of stream-stream > joins > -- > > Key: SPARK-38809 > URL: https://issues.apache.org/jira/browse/SPARK-38809 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 3.2.1 >Reporter: Anish Shrigondekar >Priority: Major > > Implement option to skip null values in symmetric hash impl of stream-stream > joins > * In the symmetric has join state manager, we can receive entries with null > values for a key and that caused the `removeByValue` and get iterators to > fail and run into the NullPointerException. > * This is possible if the state recovered is written from a old spark > version or its corrupted on disk. Since we don't have a utility to query this > state, we would like to provide a conf option to skip nulls for the symmetric > hash impl in stream stream joins. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
