[jira] [Created] (SPARK-49594) Add check on whether columnFamilies were added or removed to write StateSchemaV3 file

2024-09-10 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-49594: Summary: Add check on whether columnFamilies were added or removed to write StateSchemaV3 file Key: SPARK-49594 URL: https://issues.apache.org/jira/browse/SPARK-49594

[jira] [Updated] (SPARK-49050) Enabling deleteIfExists operator in TWS with Virtual Column Families

2024-08-27 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-49050: - Summary: Enabling deleteIfExists operator in TWS with Virtual Column Families (was: nabling del

[jira] [Updated] (SPARK-49050) nabling deleteIfExists operator in TWS with Virtual Column Families

2024-08-27 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-49050: - Description: Fully integrating the TransformWithState operator with Virtual Column Families by a

[jira] [Created] (SPARK-49050) Integrate TransformWithState operator with Virtual Column Families

2024-07-29 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-49050: Summary: Integrate TransformWithState operator with Virtual Column Families Key: SPARK-49050 URL: https://issues.apache.org/jira/browse/SPARK-49050 Project: Spark

[jira] [Created] (SPARK-49031) Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2

2024-07-26 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-49031: Summary: Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2 Key: SPARK-49031 URL: https://issues.apache.org/jira/browse/SPARK-49031

[jira] [Created] (SPARK-49006) Purge OperatorStateMetadataV2 files and StateSchemaV3 files

2024-07-25 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-49006: Summary: Purge OperatorStateMetadataV2 files and StateSchemaV3 files Key: SPARK-49006 URL: https://issues.apache.org/jira/browse/SPARK-49006 Project: Spark

[jira] [Created] (SPARK-48849) Create OperatorStateMetadataV2 for the TransformWithStateExec operator

2024-07-09 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-48849: Summary: Create OperatorStateMetadataV2 for the TransformWithStateExec operator Key: SPARK-48849 URL: https://issues.apache.org/jira/browse/SPARK-48849 Project: Spark

[jira] [Created] (SPARK-48726) Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator

2024-06-26 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-48726: Summary: Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator Key: SPARK-48726 URL: https://issues.apache.org/jira/browse/SPARK-48726

[jira] [Updated] (SPARK-47941) Propagate ForeachBatch worker initialization errors to users for PySpark

2024-04-22 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-47941: - Summary: Propagate ForeachBatch worker initialization errors to users for PySpark (was: Propaga

[jira] [Updated] (SPARK-47805) [Arbitrary State Support] State TTL support - MapState

2024-04-10 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-47805: - Description: Add support for expiring state value based on ttl for Map State in transformWithSta

[jira] [Created] (SPARK-47805) [Arbitrary State Support] State TTL support - MapState

2024-04-10 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-47805: Summary: [Arbitrary State Support] State TTL support - MapState Key: SPARK-47805 URL: https://issues.apache.org/jira/browse/SPARK-47805 Project: Spark Issue

[jira] [Created] (SPARK-47673) [Arbitrary State Support] State TTL support - ListState

2024-04-01 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-47673: Summary: [Arbitrary State Support] State TTL support - ListState Key: SPARK-47673 URL: https://issues.apache.org/jira/browse/SPARK-47673 Project: Spark Issue

[jira] [Created] (SPARK-46961) Adding processorHandle as a Context Variable

2024-02-02 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-46961: Summary: Adding processorHandle as a Context Variable Key: SPARK-46961 URL: https://issues.apache.org/jira/browse/SPARK-46961 Project: Spark Issue Type: Impr

[jira] [Created] (SPARK-46960) Testing Multiple Input Streams for TransformWithState operator

2024-02-02 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-46960: Summary: Testing Multiple Input Streams for TransformWithState operator Key: SPARK-46960 URL: https://issues.apache.org/jira/browse/SPARK-46960 Project: Spark

[jira] [Created] (SPARK-46911) Add deleteIfExists operator to StatefulProcessorHandle

2024-01-29 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-46911: Summary: Add deleteIfExists operator to StatefulProcessorHandle Key: SPARK-46911 URL: https://issues.apache.org/jira/browse/SPARK-46911 Project: Spark Issue

[jira] [Updated] (SPARK-46865) Add Batch Support for TransformWithState Operator

2024-01-25 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-46865: - Description: Add Batch support for the TransformWithState operator to maintain parity between th

[jira] [Created] (SPARK-46865) Add Batch Support for TransformWithState Operator

2024-01-25 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-46865: Summary: Add Batch Support for TransformWithState Operator Key: SPARK-46865 URL: https://issues.apache.org/jira/browse/SPARK-46865 Project: Spark Issue Type:

[jira] [Created] (SPARK-46864) Onboard Arbtirary State V2 onto New Error Class Framework

2024-01-25 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-46864: Summary: Onboard Arbtirary State V2 onto New Error Class Framework Key: SPARK-46864 URL: https://issues.apache.org/jira/browse/SPARK-46864 Project: Spark Iss

[jira] [Created] (SPARK-44480) Add option for thread pool to perform maintenance for RocksDB/HDFS State Store Providers

2023-07-18 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-44480: Summary: Add option for thread pool to perform maintenance for RocksDB/HDFS State Store Providers Key: SPARK-44480 URL: https://issues.apache.org/jira/browse/SPARK-44480

[jira] [Updated] (SPARK-44440) Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers

2023-07-14 Thread Eric Marnadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Marnadi updated SPARK-0: - Shepherd: Jungtaek Lim > Use thread pool to perform maintenance activity for hdfs/rocksdb state

[jira] [Created] (SPARK-44440) Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers

2023-07-14 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-0: Summary: Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers Key: SPARK-0 URL: https://issues.apache.org/jira/browse/SPARK-0

[jira] [Created] (SPARK-43542) Define a new error class and apply for the case where streaming query fails due to concurrent run of streaming query with same checkpoint

2023-05-17 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-43542: Summary: Define a new error class and apply for the case where streaming query fails due to concurrent run of streaming query with same checkpoint Key: SPARK-43542 URL: https://i