[jira] [Created] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

2024-05-02 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-48105: --- Summary: Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store Key: SPARK-48105 URL:

[jira] [Created] (SPARK-45942) Only do the thread interruption check for putIterator on executors

2023-11-15 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-45942: --- Summary: Only do the thread interruption check for putIterator on executors Key: SPARK-45942 URL: https://issues.apache.org/jira/browse/SPARK-45942 Project: Spark

[jira] [Updated] (SPARK-42794) Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming

2023-03-14 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42794: Description: We are seeing query failure which is caused by RocksDB acquisition failure for the

[jira] [Updated] (SPARK-42794) Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming

2023-03-14 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42794: Priority: Minor (was: Major) > Increase the lockAcquireTimeoutMs for acquiring the RocksDB state

[jira] [Updated] (SPARK-42794) Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming

2023-03-14 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42794: Description: We are seeing query failure which is caused by RocksDB acquisition failure for the

[jira] [Updated] (SPARK-42794) Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming

2023-03-14 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42794: Description: We are seeing query failure which is caused by RocksDB acquisition failure for the

[jira] [Created] (SPARK-42794) Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming

2023-03-14 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-42794: --- Summary: Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming Key: SPARK-42794 URL: https://issues.apache.org/jira/browse/SPARK-42794

[jira] [Created] (SPARK-42687) Better error message for unspported `Pivot` operator in Structure Streaming

2023-03-06 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-42687: --- Summary: Better error message for unspported `Pivot` operator in Structure Streaming Key: SPARK-42687 URL: https://issues.apache.org/jira/browse/SPARK-42687 Project:

[jira] [Updated] (SPARK-42566) RocksDB StateStore lock acquisition should happen after getting input iterator from inputRDD

2023-02-24 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42566: Summary: RocksDB StateStore lock acquisition should happen after getting input iterator from

[jira] [Created] (SPARK-42566) RocksDB StateStore lock acquisition should happen before getting input iterator from inputRDD

2023-02-24 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-42566: --- Summary: RocksDB StateStore lock acquisition should happen before getting input iterator from inputRDD Key: SPARK-42566 URL: https://issues.apache.org/jira/browse/SPARK-42566

[jira] [Updated] (SPARK-42565) Error log improve ment for the lock acquisition of RocksDB state store instance

2023-02-24 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42565: Summary: Error log improve ment for the lock acquisition of RocksDB state store instance (was:

[jira] [Updated] (SPARK-42565) Error log improvement for RocksDB state store instance lock acquisition

2023-02-24 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-42565: Description:   {code:java} "23/02/23 23:57:44 INFO Executor: Running task 2.0 in stage 57.1 (TID

[jira] [Created] (SPARK-42565) Error log improvement for RocksDB state store instance lock acquisition

2023-02-24 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-42565: --- Summary: Error log improvement for RocksDB state store instance lock acquisition Key: SPARK-42565 URL: https://issues.apache.org/jira/browse/SPARK-42565 Project: Spark

[jira] [Updated] (SPARK-41524) Expose SQL confs and extraOptions separately in o.a.s.sql.execution.streaming.state.RocksDBConf

2022-12-14 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-41524: Priority: Minor (was: Major) > Expose SQL confs and extraOptions separately in >

[jira] [Created] (SPARK-41524) Expose SQL confs and extraOptions separately in o.a.s.sql.execution.streaming.state.RocksDBConf

2022-12-14 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-41524: --- Summary: Expose SQL confs and extraOptions separately in o.a.s.sql.execution.streaming.state.RocksDBConf Key: SPARK-41524 URL: https://issues.apache.org/jira/browse/SPARK-41524

[jira] [Updated] (SPARK-40466) Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available

2022-09-15 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-40466: Shepherd: (was: Huanli Wang) > Improve the error message if the DSv2 source is disabled but

[jira] [Updated] (SPARK-40466) Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available

2022-09-15 Thread Huanli Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huanli Wang updated SPARK-40466: Shepherd: Huanli Wang > Improve the error message if the DSv2 source is disabled but DSv1

[jira] [Created] (SPARK-40466) Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available

2022-09-15 Thread Huanli Wang (Jira)
Huanli Wang created SPARK-40466: --- Summary: Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available Key: SPARK-40466 URL: https://issues.apache.org/jira/browse/SPARK-40466