[jira] [Resolved] (SPARK-48799) Support versioning for operator metadata reader/writer and update callers

[ https://issues.apache.org/jira/browse/SPARK-48799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48799. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47203

[jira] [Assigned] (SPARK-48799) Support versioning for operator metadata reader/writer and update callers

[ https://issues.apache.org/jira/browse/SPARK-48799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48799: Assignee: Anish Shrigondekar > Support versioning for operator metadata reader/writer

[jira] [Resolved] (SPARK-48770) Read operator metadata only once on driver for store read

[ https://issues.apache.org/jira/browse/SPARK-48770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48770. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47167

[jira] [Assigned] (SPARK-48770) Read operator metadata only once on driver for store read

[ https://issues.apache.org/jira/browse/SPARK-48770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48770: Assignee: Anish Shrigondekar > Read operator metadata only once on driver for store read

[jira] [Assigned] (SPARK-48589) Add option snapshotStartBatchId and snapshotPartitionId to state data source

[ https://issues.apache.org/jira/browse/SPARK-48589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48589: Assignee: Yuchen Liu > Add option snapshotStartBatchId and snapshotPartitionId to state

[jira] [Resolved] (SPARK-48589) Add option snapshotStartBatchId and snapshotPartitionId to state data source

[ https://issues.apache.org/jira/browse/SPARK-48589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48589. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46944

[jira] [Updated] (SPARK-48586) Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

[ https://issues.apache.org/jira/browse/SPARK-48586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48586: - Fix Version/s: 3.5.2 > Remove lock acquisition in doMaintenance() by making a deep copy of file

[jira] [Resolved] (SPARK-48586) Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

[ https://issues.apache.org/jira/browse/SPARK-48586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48586. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved via

[jira] [Assigned] (SPARK-48586) Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

[ https://issues.apache.org/jira/browse/SPARK-48586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48586: Assignee: Riya Verma > Remove lock acquisition in doMaintenance() by making a deep copy

[jira] [Assigned] (SPARK-48687) Add changes to implement state schema validation in planning phase on driver for stateful streaming queries

[ https://issues.apache.org/jira/browse/SPARK-48687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48687: Assignee: Anish Shrigondekar > Add changes to implement state schema validation in

[jira] [Resolved] (SPARK-48687) Add changes to implement state schema validation in planning phase on driver for stateful streaming queries

[ https://issues.apache.org/jira/browse/SPARK-48687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48687. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47035

[jira] [Resolved] (SPARK-48543) Track invalid unsafe row exception explicitly as error class

[ https://issues.apache.org/jira/browse/SPARK-48543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48543. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46885

[jira] [Assigned] (SPARK-48543) Track invalid unsafe row exception explicitly as error class

[ https://issues.apache.org/jira/browse/SPARK-48543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48543: Assignee: Anish Shrigondekar > Track invalid unsafe row exception explicitly as error

[jira] [Created] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan

Jungtaek Lim created SPARK-48597: Summary: Distinguish the streaming nodes from the text representation of logical plan Key: SPARK-48597 URL: https://issues.apache.org/jira/browse/SPARK-48597

[jira] [Commented] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan

[ https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17854291#comment-17854291 ] Jungtaek Lim commented on SPARK-48597: -- Will submit a PR sooner. > Distinguish the streaming nodes

[jira] [Resolved] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48411. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46740

[jira] [Assigned] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48411: Assignee: Yuchen Liu > Add E2E test for DropDuplicateWithinWatermark >

[jira] [Resolved] (SPARK-48513) Use NERF framework for state schema compatibility exception

[ https://issues.apache.org/jira/browse/SPARK-48513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48513. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46856

[jira] [Assigned] (SPARK-48513) Use NERF framework for state schema compatibility exception

[ https://issues.apache.org/jira/browse/SPARK-48513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48513: Assignee: Anish Shrigondekar > Use NERF framework for state schema compatibility

[jira] [Resolved] (SPARK-48447) Check state store provider class before invoking the constructor

[ https://issues.apache.org/jira/browse/SPARK-48447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48447. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46791

[jira] [Assigned] (SPARK-48447) Check state store provider class before invoking the constructor

[ https://issues.apache.org/jira/browse/SPARK-48447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48447: Assignee: Yuchen Liu > Check state store provider class before invoking the constructor

[jira] [Resolved] (SPARK-48383) In KafkaOffsetReader, partition mismatch should not be an assertion

[ https://issues.apache.org/jira/browse/SPARK-48383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48383. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46692

[jira] [Assigned] (SPARK-48383) In KafkaOffsetReader, partition mismatch should not be an assertion

[ https://issues.apache.org/jira/browse/SPARK-48383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48383: Assignee: Siying Dong > In KafkaOffsetReader, partition mismatch should not be an

[jira] [Resolved] (SPARK-47920) Add documentation for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47920. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46139

[jira] [Resolved] (SPARK-48314) FileStreamSource shouldn't double cache files for availableNow

[ https://issues.apache.org/jira/browse/SPARK-48314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48314. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46627

[jira] [Assigned] (SPARK-48314) FileStreamSource shouldn't double cache files for availableNow

[ https://issues.apache.org/jira/browse/SPARK-48314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48314: Assignee: Adam Binford > FileStreamSource shouldn't double cache files for availableNow

[jira] [Assigned] (SPARK-48330) Fix the python streaming data source timeout issue for large trigger interval

[ https://issues.apache.org/jira/browse/SPARK-48330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48330: Assignee: Chaoqin Li > Fix the python streaming data source timeout issue for large

[jira] [Resolved] (SPARK-48330) Fix the python streaming data source timeout issue for large trigger interval

[ https://issues.apache.org/jira/browse/SPARK-48330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48330. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46651

[jira] [Assigned] (SPARK-44924) Add configurations for FileStreamSource cached files

[ https://issues.apache.org/jira/browse/SPARK-44924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-44924: Assignee: kevin nacios > Add configurations for FileStreamSource cached files >

[jira] [Resolved] (SPARK-44924) Add configurations for FileStreamSource cached files

[ https://issues.apache.org/jira/browse/SPARK-44924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-44924. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45362

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Labels: correctness pull-request-available (was: pull-request-available) > Fix the data

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Issue Type: Bug (was: Improvement) > Fix the data corruption issue when state store unload

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Priority: Blocker (was: Major) > Fix the data corruption issue when state store unload and

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Fix Version/s: 3.4.4 > Fix the data corruption issue when state store unload and snapshotting

[jira] [Resolved] (SPARK-48293) Add test for when ForeachBatchUserFuncException wraps interrupted exception due to query stop

[ https://issues.apache.org/jira/browse/SPARK-48293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48293. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46601

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Fix Version/s: 3.5.2 > Fix the data corruption issue when state store unload and snapshotting

[jira] [Resolved] (SPARK-48233) Tests for non-stateful streaming with collations

[ https://issues.apache.org/jira/browse/SPARK-48233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48233. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46247

[jira] [Updated] (SPARK-48267) Regression e2e test with SPARK-47305

[ https://issues.apache.org/jira/browse/SPARK-48267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48267: - Fix Version/s: 3.5.2 > Regression e2e test with SPARK-47305 >

[jira] [Resolved] (SPARK-48267) Regression e2e test with SPARK-47305

[ https://issues.apache.org/jira/browse/SPARK-48267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48267. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46569

[jira] [Assigned] (SPARK-48267) Regression e2e test with SPARK-47305

[ https://issues.apache.org/jira/browse/SPARK-48267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48267: Assignee: Jungtaek Lim > Regression e2e test with SPARK-47305 >

[jira] [Created] (SPARK-48267) Regression e2e test with SPARK-47305

Jungtaek Lim created SPARK-48267: Summary: Regression e2e test with SPARK-47305 Key: SPARK-48267 URL: https://issues.apache.org/jira/browse/SPARK-48267 Project: Spark Issue Type: Test

[jira] [Assigned] (SPARK-48208) Skip reporting memory usage metrics if bounded memory usage is enabled

[ https://issues.apache.org/jira/browse/SPARK-48208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48208: Assignee: Anish Shrigondekar > Skip reporting memory usage metrics if bounded memory

[jira] [Resolved] (SPARK-48208) Skip reporting memory usage metrics if bounded memory usage is enabled

[ https://issues.apache.org/jira/browse/SPARK-48208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48208. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46491

[jira] [Assigned] (SPARK-47960) Support Chaining Stateful Operators in TransformWithState

[ https://issues.apache.org/jira/browse/SPARK-47960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47960: Assignee: Bhuwan Sahni > Support Chaining Stateful Operators in TransformWithState >

[jira] [Resolved] (SPARK-47960) Support Chaining Stateful Operators in TransformWithState

[ https://issues.apache.org/jira/browse/SPARK-47960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47960. -- Resolution: Fixed Issue resolved by pull request 45376

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48105: - Affects Version/s: 3.5.2 3.4.4 > Fix the data corruption issue when

[jira] [Resolved] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48105. -- Fix Version/s: 4.0.0 Assignee: Huanli Wang Resolution: Fixed Issue resolved

[jira] [Resolved] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

[ https://issues.apache.org/jira/browse/SPARK-48102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48102. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46350

[jira] [Assigned] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

[ https://issues.apache.org/jira/browse/SPARK-48102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48102: Assignee: Anish Shrigondekar > Track time to acquire source progress metrics for

[jira] [Updated] (SPARK-47920) Add documentation for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-47920: - Affects Version/s: 4.0.0 (was: 3.5.1) > Add documentation for python

[jira] [Commented] (SPARK-48073) StateStore schema incompatibility between 3.2 and 3.4

[ https://issues.apache.org/jira/browse/SPARK-48073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842785#comment-17842785 ] Jungtaek Lim commented on SPARK-48073: -- I roughly remember that Encoder.bean() had changed to

[jira] [Resolved] (SPARK-47793) Implement SimpleDataSourceStreamReader for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47793. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45977

[jira] [Assigned] (SPARK-47793) Implement SimpleDataSourceStreamReader for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47793: Assignee: Chaoqin Li > Implement SimpleDataSourceStreamReader for python streaming data

[jira] [Assigned] (SPARK-48050) Log logical plan at query start

[ https://issues.apache.org/jira/browse/SPARK-48050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48050: Assignee: Fanyue Xia > Log logical plan at query start > ---

[jira] [Resolved] (SPARK-48050) Log logical plan at query start

[ https://issues.apache.org/jira/browse/SPARK-48050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48050. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46292

[jira] [Assigned] (SPARK-48018) Null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange

[ https://issues.apache.org/jira/browse/SPARK-48018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48018: Assignee: B. Micheal Okutubo > Null groupId causing missing param error when throwing >

[jira] [Resolved] (SPARK-48018) Null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange

[ https://issues.apache.org/jira/browse/SPARK-48018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48018. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46253

[jira] [Resolved] (SPARK-47805) [Arbitrary State Support] State TTL support - MapState

[ https://issues.apache.org/jira/browse/SPARK-47805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47805. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45991

[jira] [Assigned] (SPARK-47840) Remove foldable propagation across Streaming Aggregate/Join nodes

[ https://issues.apache.org/jira/browse/SPARK-47840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47840: Assignee: Bhuwan Sahni > Remove foldable propagation across Streaming Aggregate/Join

[jira] [Resolved] (SPARK-47840) Remove foldable propagation across Streaming Aggregate/Join nodes

[ https://issues.apache.org/jira/browse/SPARK-47840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47840. -- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-47673) [Arbitrary State Support] State TTL support - ListState

[ https://issues.apache.org/jira/browse/SPARK-47673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47673. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45932

[jira] [Resolved] (SPARK-47788) Ensure the same hash partitioning scheme/hash function is used across batches

[ https://issues.apache.org/jira/browse/SPARK-47788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47788. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45971

[jira] [Assigned] (SPARK-47788) Ensure the same hash partitioning scheme/hash function is used across batches

[ https://issues.apache.org/jira/browse/SPARK-47788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47788: Assignee: Fanyue Xia > Ensure the same hash partitioning scheme/hash function is used

[jira] [Resolved] (SPARK-47848) Fix thread safe access for loadedMaps in close

[ https://issues.apache.org/jira/browse/SPARK-47848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47848. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46048

[jira] [Assigned] (SPARK-47733) Add operational metrics for TWS operators

[ https://issues.apache.org/jira/browse/SPARK-47733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47733: Assignee: Anish Shrigondekar (was: Jing Zhan) > Add operational metrics for TWS

[jira] [Assigned] (SPARK-47733) Add operational metrics for TWS operators

[ https://issues.apache.org/jira/browse/SPARK-47733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47733: Assignee: Jing Zhan > Add operational metrics for TWS operators >

[jira] [Resolved] (SPARK-47784) [State API v2] Merge TimeoutMode and TTLMode into TimeMode

[ https://issues.apache.org/jira/browse/SPARK-47784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47784. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45960

[jira] [Assigned] (SPARK-47776) State store operation cannot work properly with binary inequality collation

[ https://issues.apache.org/jira/browse/SPARK-47776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47776: Assignee: Jungtaek Lim > State store operation cannot work properly with binary

[jira] [Resolved] (SPARK-47776) State store operation cannot work properly with binary inequality collation

[ https://issues.apache.org/jira/browse/SPARK-47776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47776. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45951

[jira] [Commented] (SPARK-47718) .sql() does not recognize watermark defined upstream

[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835303#comment-17835303 ] Jungtaek Lim commented on SPARK-47718: -- window('createTime', '1 hour', '30 minutes') 'createTime'

[jira] [Resolved] (SPARK-47718) .sql() does not recognize watermark defined upstream

[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47718. -- Resolution: Not A Bug > .sql() does not recognize watermark defined upstream >

[jira] [Created] (SPARK-47776) State store operation cannot work properly with binary inequality collation

Jungtaek Lim created SPARK-47776: Summary: State store operation cannot work properly with binary inequality collation Key: SPARK-47776 URL: https://issues.apache.org/jira/browse/SPARK-47776 Project:

[jira] [Commented] (SPARK-47718) .sql() does not recognize watermark defined upstream

[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835175#comment-17835175 ] Jungtaek Lim commented on SPARK-47718: -- I've lowered down to major - this is neither a regression

[jira] [Updated] (SPARK-47718) .sql() does not recognize watermark defined upstream

[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-47718: - Priority: Major (was: Blocker) > .sql() does not recognize watermark defined upstream >

[jira] [Resolved] (SPARK-47746) Use column ordinals instead of prefix ordering columns in the range scan encoder

[ https://issues.apache.org/jira/browse/SPARK-47746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47746. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45905

[jira] [Resolved] (SPARK-47558) [Arbitrary State Support] State TTL support - ValueState

[ https://issues.apache.org/jira/browse/SPARK-47558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47558. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45674

[jira] [Assigned] (SPARK-47299) Use the same `versions. json` in the dropdown of different versions of PySpark documents

[ https://issues.apache.org/jira/browse/SPARK-47299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47299: Assignee: BingKun Pan > Use the same `versions. json` in the dropdown of different

[jira] [Resolved] (SPARK-47299) Use the same `versions. json` in the dropdown of different versions of PySpark documents

[ https://issues.apache.org/jira/browse/SPARK-47299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47299. -- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-47734) Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

[ https://issues.apache.org/jira/browse/SPARK-47734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-47734: - Fix Version/s: 3.4.3 > Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by

[jira] [Resolved] (SPARK-47744) Add support for negative byte types in RocksDB range scan encoder

[ https://issues.apache.org/jira/browse/SPARK-47744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47744. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45906

[jira] [Assigned] (SPARK-47744) Add support for negative byte types in RocksDB range scan encoder

[ https://issues.apache.org/jira/browse/SPARK-47744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47744: Assignee: Neil Ramaswamy > Add support for negative byte types in RocksDB range scan

[jira] [Resolved] (SPARK-47653) Add support for negative numbers with range scan encoder

[ https://issues.apache.org/jira/browse/SPARK-47653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47653. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45778

[jira] [Resolved] (SPARK-47655) Integrate timer with Initial State handling for state-v2

[ https://issues.apache.org/jira/browse/SPARK-47655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47655. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45780

[jira] [Resolved] (SPARK-47568) Fix race condition between maintenance thread and task thead for RocksDB snapshot

[ https://issues.apache.org/jira/browse/SPARK-47568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47568. -- Fix Version/s: 4.0.0 Assignee: Bhuwan Sahni Resolution: Fixed Issue resolved

[jira] [Resolved] (SPARK-47363) Initial State without state reader implementation for State API v2.

[ https://issues.apache.org/jira/browse/SPARK-47363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47363. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45467

[jira] [Resolved] (SPARK-47107) Implement partition reader for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47107. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45485

[jira] [Assigned] (SPARK-47107) Implement partition reader for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47107: Assignee: Chaoqin Li > Implement partition reader for python streaming data source >

[jira] [Assigned] (SPARK-47570) Integrate range scan encoder changes with timer implementation

[ https://issues.apache.org/jira/browse/SPARK-47570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47570: Assignee: Jing Zhan > Integrate range scan encoder changes with timer implementation >

[jira] [Resolved] (SPARK-47570) Integrate range scan encoder changes with timer implementation

[ https://issues.apache.org/jira/browse/SPARK-47570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47570. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45709

[jira] [Resolved] (SPARK-47273) Implement python stream writer interface

[ https://issues.apache.org/jira/browse/SPARK-47273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47273. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45305

[jira] [Assigned] (SPARK-47273) Implement python stream writer interface

[ https://issues.apache.org/jira/browse/SPARK-47273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47273: Assignee: Chaoqin Li > Implement python stream writer interface >

[jira] [Resolved] (SPARK-47512) Tag operation type for RocksDB instance lock acquisition

[ https://issues.apache.org/jira/browse/SPARK-47512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47512. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45651

[jira] [Resolved] (SPARK-47449) Refactor and split list/timer unit tests

[ https://issues.apache.org/jira/browse/SPARK-47449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47449. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45573

[jira] [Resolved] (SPARK-47329) Persist df while using foreachbatch and stateful streaming query to prevent state from being re-loaded in each batch

[ https://issues.apache.org/jira/browse/SPARK-47329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47329. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45432

[jira] [Resolved] (SPARK-46913) Implement timer functionality for transformWithState operator

[ https://issues.apache.org/jira/browse/SPARK-46913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46913. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45051

[jira] [Resolved] (SPARK-47272) MapState Implementation for State V2

[ https://issues.apache.org/jira/browse/SPARK-47272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47272. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45341

[jira] [Assigned] (SPARK-46962) Implement python worker to run python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-46962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-46962: Assignee: Chaoqin Li > Implement python worker to run python streaming data source >

[jira] [Resolved] (SPARK-46962) Implement python worker to run python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-46962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46962. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45023

[jira] [Resolved] (SPARK-47331) Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2.

[ https://issues.apache.org/jira/browse/SPARK-47331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47331. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45447

[jira] [Assigned] (SPARK-47331) Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2.

[ https://issues.apache.org/jira/browse/SPARK-47331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47331: Assignee: Jing Zhan > Serialization using case classes/primitives/POJO based on SQL

  1   2   3   4   5   6   7   8   9   10   >