[jira] [Resolved] (SPARK-52391) Refactor TransformWithStateExec to extract shared functions and variables into an abstract base class for Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-52391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52391. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51077 [https://gi

[jira] [Assigned] (SPARK-52391) Refactor TransformWithStateExec to extract shared functions and variables into an abstract base class for Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-52391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52391: Assignee: Huanli Wang > Refactor TransformWithStateExec to extract shared functions and v

[jira] [Assigned] (SPARK-52350) Fix link for SS programming guide page

[ https://issues.apache.org/jira/browse/SPARK-52350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52350: Assignee: Anish Shrigondekar > Fix link for SS programming guide page > -

[jira] [Resolved] (SPARK-52350) Fix link for SS programming guide page

[ https://issues.apache.org/jira/browse/SPARK-52350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52350. -- Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

[ https://issues.apache.org/jira/browse/SPARK-52333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52333: Assignee: Jungtaek Lim > Squeeze protocol for timers (list on specific grouping key, and

[jira] [Resolved] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

[ https://issues.apache.org/jira/browse/SPARK-52333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52333. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51036 [https://gi

[jira] [Resolved] (SPARK-52228) Introduce the benchmark setup (manual) for state interaction between TWS state server and Python process

[ https://issues.apache.org/jira/browse/SPARK-52228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52228. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50952 [https://gi

[jira] [Assigned] (SPARK-52228) Introduce the benchmark setup (manual) for state interaction between TWS state server and Python process

[ https://issues.apache.org/jira/browse/SPARK-52228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52228: Assignee: Jungtaek Lim > Introduce the benchmark setup (manual) for state interaction bet

[jira] [Commented] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

[ https://issues.apache.org/jira/browse/SPARK-52333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954468#comment-17954468 ] Jungtaek Lim commented on SPARK-52333: -- Going to submit a PR for this. Probably in

[jira] [Created] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

Jungtaek Lim created SPARK-52333: Summary: Squeeze protocol for timers (list on specific grouping key, and expiry timers) Key: SPARK-52333 URL: https://issues.apache.org/jira/browse/SPARK-52333 Projec

[jira] [Resolved] (SPARK-52329) Remove private[sql] tags for new transformWithState API

[ https://issues.apache.org/jira/browse/SPARK-52329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52329. -- Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-52329) Remove private[sql] tags for new transformWithState API

[ https://issues.apache.org/jira/browse/SPARK-52329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52329: Assignee: Anish Shrigondekar > Remove private[sql] tags for new transformWithState API >

[jira] [Resolved] (SPARK-52195) Fix initial state column dropping issue for Python TWS

[ https://issues.apache.org/jira/browse/SPARK-52195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52195. -- Fix Version/s: 4.1.0 Assignee: Bo Gao Resolution: Fixed Issue resolved via [ht

[jira] [Created] (SPARK-52228) Introduce the benchmark setup (manual) for state interaction between TWS state server and Python process

Jungtaek Lim created SPARK-52228: Summary: Introduce the benchmark setup (manual) for state interaction between TWS state server and Python process Key: SPARK-52228 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Resolved] (SPARK-52188) State store source can't read from RocksDB

[ https://issues.apache.org/jira/browse/SPARK-52188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52188. -- Fix Version/s: 4.0.0 4.1.0 Assignee: Eric Marnadi Resolution

[jira] [Resolved] (SPARK-52096) Reclassify kafka source offset assertion error

[ https://issues.apache.org/jira/browse/SPARK-52096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52096. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50866 [https://gi

[jira] [Assigned] (SPARK-52096) Reclassify kafka source offset assertion error

[ https://issues.apache.org/jira/browse/SPARK-52096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52096: Assignee: Yuchen Liu > Reclassify kafka source offset assertion error > -

[jira] [Assigned] (SPARK-52126) Revert rename for TWS utility classes for forward compatibility in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-52126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52126: Assignee: Jungtaek Lim > Revert rename for TWS utility classes for forward compatibility

[jira] [Resolved] (SPARK-52126) Revert rename for TWS utility classes for forward compatibility in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-52126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52126. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50883 [https://gi

[jira] [Assigned] (SPARK-52126) Revert rename for TWS utility classes for forward compatibility in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-52126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52126: Assignee: (was: Jungtaek Lim) > Revert rename for TWS utility classes for forward com

[jira] [Assigned] (SPARK-52126) Revert rename for TWS utility classes for forward compatibility in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-52126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52126: Assignee: Jungtaek Lim > Revert rename for TWS utility classes for forward compatibility

[jira] [Created] (SPARK-52126) Revert rename for TWS utility classes for forward compatibility in Spark Connect

Jungtaek Lim created SPARK-52126: Summary: Revert rename for TWS utility classes for forward compatibility in Spark Connect Key: SPARK-52126 URL: https://issues.apache.org/jira/browse/SPARK-52126 Proj

[jira] [Created] (SPARK-52065) Produce another plan tree with output columns (name, data type, nullability) in plan change logging

Jungtaek Lim created SPARK-52065: Summary: Produce another plan tree with output columns (name, data type, nullability) in plan change logging Key: SPARK-52065 URL: https://issues.apache.org/jira/browse/SPARK-5206

[jira] [Assigned] (SPARK-51291) Reclassify Validation Errors from CANNOT_LOAD_STATE_STORE.UNCATEGORIZED

[ https://issues.apache.org/jira/browse/SPARK-51291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51291: Assignee: Livia Zhu > Reclassify Validation Errors from CANNOT_LOAD_STATE_STORE.UNCATEGOR

[jira] [Resolved] (SPARK-51291) Reclassify Validation Errors from CANNOT_LOAD_STATE_STORE.UNCATEGORIZED

[ https://issues.apache.org/jira/browse/SPARK-51291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51291. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50045 [https://gi

[jira] [Assigned] (SPARK-51940) Support an interface for checkpoint log metadata management

[ https://issues.apache.org/jira/browse/SPARK-51940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51940: Assignee: Jackie Zhang > Support an interface for checkpoint log metadata management > --

[jira] [Resolved] (SPARK-51940) Support an interface for checkpoint log metadata management

[ https://issues.apache.org/jira/browse/SPARK-51940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51940. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50746 [https://gi

[jira] [Resolved] (SPARK-51933) Document the new API `transformWithState` in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51933. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50738 [https://gi

[jira] [Resolved] (SPARK-51869) Create classification for user errors within handleInputRows for Scala TransformWithState

[ https://issues.apache.org/jira/browse/SPARK-51869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51869. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50667 [https://gi

[jira] [Assigned] (SPARK-51869) Create classification for user errors within handleInputRows for Scala TransformWithState

[ https://issues.apache.org/jira/browse/SPARK-51869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51869: Assignee: Eric Marnadi > Create classification for user errors within handleInputRows for

[jira] [Created] (SPARK-51933) Document the new API `transformWithState` in PySpark

Jungtaek Lim created SPARK-51933: Summary: Document the new API `transformWithState` in PySpark Key: SPARK-51933 URL: https://issues.apache.org/jira/browse/SPARK-51933 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-51827) Support Spark Connect in new API `transformWithState` in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51827. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50704 [https://gi

[jira] [Assigned] (SPARK-51827) Support Spark Connect in new API `transformWithState` in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51827: Assignee: Jungtaek Lim > Support Spark Connect in new API `transformWithState` in PySpark

[jira] [Assigned] (SPARK-51823) Add option to not persist state stores on executors

[ https://issues.apache.org/jira/browse/SPARK-51823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51823: Assignee: Adam Binford > Add option to not persist state stores on executors > --

[jira] [Resolved] (SPARK-51823) Add option to not persist state stores on executors

[ https://issues.apache.org/jira/browse/SPARK-51823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51823. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50612 [https://gi

[jira] [Resolved] (SPARK-51827) Support Spark Connect in new API `transformWithState` in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51827. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50704 [https://gi

[jira] [Assigned] (SPARK-51827) Support Spark Connect in new API `transformWithState` in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51827: Assignee: Jungtaek Lim > Support Spark Connect in new API `transformWithState` in PySpark

[jira] [Resolved] (SPARK-51922) Fix UTFDataFormatException thrown from StateStoreChangelogReaderFactory for v1

[ https://issues.apache.org/jira/browse/SPARK-51922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51922. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50721 [https://gi

[jira] [Assigned] (SPARK-51922) Fix UTFDataFormatException thrown from StateStoreChangelogReaderFactory for v1

[ https://issues.apache.org/jira/browse/SPARK-51922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51922: Assignee: Livia Zhu > Fix UTFDataFormatException thrown from StateStoreChangelogReaderFac

[jira] [Resolved] (SPARK-51904) Removing async metadata purging for OperatorStateMetadataV2

[ https://issues.apache.org/jira/browse/SPARK-51904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51904. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50700 [https://gi

[jira] [Assigned] (SPARK-51904) Removing async metadata purging for OperatorStateMetadataV2

[ https://issues.apache.org/jira/browse/SPARK-51904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51904: Assignee: Eric Marnadi > Removing async metadata purging for OperatorStateMetadataV2 > --

[jira] [Assigned] (SPARK-51891) Squeeze the protocol of ListState GET / PUT / APPENDLIST for transformWithState in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51891: Assignee: Jungtaek Lim > Squeeze the protocol of ListState GET / PUT / APPENDLIST for >

[jira] [Resolved] (SPARK-51891) Squeeze the protocol of ListState GET / PUT / APPENDLIST for transformWithState in PySpark

[ https://issues.apache.org/jira/browse/SPARK-51891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51891. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50689 [https://gi

[jira] [Resolved] (SPARK-51889) Fix MapState bug on clear() for TWS Python

[ https://issues.apache.org/jira/browse/SPARK-51889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51889. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50686 [https://gi

[jira] [Created] (SPARK-51891) Squeeze the protocol of ListState GET / PUT / APPENDLIST for transformWithState in PySpark

Jungtaek Lim created SPARK-51891: Summary: Squeeze the protocol of ListState GET / PUT / APPENDLIST for transformWithState in PySpark Key: SPARK-51891 URL: https://issues.apache.org/jira/browse/SPARK-51891

[jira] [Assigned] (SPARK-51822) Throwing classified error when disallowed functions are called during StatefulProcessor.init()

[ https://issues.apache.org/jira/browse/SPARK-51822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51822: Assignee: Eric Marnadi > Throwing classified error when disallowed functions are called d

[jira] [Resolved] (SPARK-51822) Throwing classified error when disallowed functions are called during StatefulProcessor.init()

[ https://issues.apache.org/jira/browse/SPARK-51822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51822. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50611 [https://gi

[jira] [Resolved] (SPARK-51779) Use virtual column families for stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-51779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51779. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50572 [https://gi

[jira] [Created] (SPARK-51827) Support Spark Connect in new API `transformWithState` in PySpark

Jungtaek Lim created SPARK-51827: Summary: Support Spark Connect in new API `transformWithState` in PySpark Key: SPARK-51827 URL: https://issues.apache.org/jira/browse/SPARK-51827 Project: Spark

[jira] [Resolved] (SPARK-51768) Create Failure Injection Test for Streaming offset and commit log write failures

[ https://issues.apache.org/jira/browse/SPARK-51768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51768. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50559 [https://gi

[jira] [Assigned] (SPARK-51768) Create Failure Injection Test for Streaming offset and commit log write failures

[ https://issues.apache.org/jira/browse/SPARK-51768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51768: Assignee: Siying Dong > Create Failure Injection Test for Streaming offset and commit log

[jira] [Resolved] (SPARK-51358) Introduce snapshot upload lag detection through StateStoreCoordinator

[ https://issues.apache.org/jira/browse/SPARK-51358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51358. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50123 [https://gi

[jira] [Assigned] (SPARK-51358) Introduce snapshot upload lag detection through StateStoreCoordinator

[ https://issues.apache.org/jira/browse/SPARK-51358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51358: Assignee: Zeyu Chen > Introduce snapshot upload lag detection through StateStoreCoordinat

[jira] [Resolved] (SPARK-51714) Add Failure Ingestion test to test state store checkpoint format V2

[ https://issues.apache.org/jira/browse/SPARK-51714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51714. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50508 [https://gi

[jira] [Assigned] (SPARK-51717) Possible SST mismatch error for the second snapshot created for a new query

[ https://issues.apache.org/jira/browse/SPARK-51717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51717: Assignee: B. Micheal Okutubo > Possible SST mismatch error for the second snapshot create

[jira] [Resolved] (SPARK-51717) Possible SST mismatch error for the second snapshot created for a new query

[ https://issues.apache.org/jira/browse/SPARK-51717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51717. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50512 [https://gi

[jira] [Resolved] (SPARK-51724) RocksDB StateStore's lineage manager should be synchronized

[ https://issues.apache.org/jira/browse/SPARK-51724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51724. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50520 [https://gi

[jira] [Assigned] (SPARK-51724) RocksDB StateStore's lineage manager should be synchronized

[ https://issues.apache.org/jira/browse/SPARK-51724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51724: Assignee: Siying Dong > RocksDB StateStore's lineage manager should be synchronized > ---

[jira] [Resolved] (SPARK-51690) Change the protocol of ListState.put()/get()/appendList() from Arrow to simple custom protocol

[ https://issues.apache.org/jira/browse/SPARK-51690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51690. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50488 [https://gi

[jira] [Resolved] (SPARK-51586) Kafka continuous stream may go into infinite loop of reconfiguring

[ https://issues.apache.org/jira/browse/SPARK-51586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51586. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50348 [https://gi

[jira] [Resolved] (SPARK-51675) Fix issue around creating col family after db open to avoid redundant snapshot creation

[ https://issues.apache.org/jira/browse/SPARK-51675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51675. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50471 [https://gi

[jira] [Resolved] (SPARK-51682) State Store Checkpoint V2 should handle offset log ahead of commit log correctly

[ https://issues.apache.org/jira/browse/SPARK-51682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51682. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50480 [https://gi

[jira] [Assigned] (SPARK-51685) Excessive Info logging from RocksDb operations causing too big executor stderr files

[ https://issues.apache.org/jira/browse/SPARK-51685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51685: Assignee: Vinod KC > Excessive Info logging from RocksDb operations causing too big execu

[jira] [Resolved] (SPARK-51667) [TWS + Python] Disable Nagle's algorithm between Python worker and State Server

[ https://issues.apache.org/jira/browse/SPARK-51667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51667. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50460 [https://gi

[jira] [Resolved] (SPARK-51685) Excessive Info logging from RocksDb operations causing too big executor stderr files

[ https://issues.apache.org/jira/browse/SPARK-51685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51685. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50483 [https://gi

[jira] [Resolved] (SPARK-51700) Fix log entry around file deletion in RocksDBFileManager

[ https://issues.apache.org/jira/browse/SPARK-51700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51700. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50502 [https://gi

[jira] [Assigned] (SPARK-51675) Fix issue around creating col family after db open to avoid redundant snapshot creation

[ https://issues.apache.org/jira/browse/SPARK-51675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51675: Assignee: Anish Shrigondekar > Fix issue around creating col family after db open to avoi

[jira] [Assigned] (SPARK-51667) [TWS + Python] Disable Nagle's algorithm between Python worker and State Server

[ https://issues.apache.org/jira/browse/SPARK-51667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51667: Assignee: Jungtaek Lim > [TWS + Python] Disable Nagle's algorithm between Python worker a

[jira] [Created] (SPARK-51667) [TWS + Python] Disable Nagle's algorithm between Python worker and State Server

Jungtaek Lim created SPARK-51667: Summary: [TWS + Python] Disable Nagle's algorithm between Python worker and State Server Key: SPARK-51667 URL: https://issues.apache.org/jira/browse/SPARK-51667 Proje

[jira] [Updated] (SPARK-50303) Enable QUERY_TAG for SQL Session in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-50303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-50303: - Priority: Major (was: Critical) > Enable QUERY_TAG for SQL Session in Spark SQL > -

[jira] [Assigned] (SPARK-51573) Fix Streaming State Checkpoint v2 checkpointInfo race condition

[ https://issues.apache.org/jira/browse/SPARK-51573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51573: Assignee: Livia Zhu > Fix Streaming State Checkpoint v2 checkpointInfo race condition > -

[jira] [Resolved] (SPARK-51573) Fix Streaming State Checkpoint v2 checkpointInfo race condition

[ https://issues.apache.org/jira/browse/SPARK-51573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51573. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50344 [https://gi

[jira] [Updated] (SPARK-51587) [PySpark] Fix an issue where timestamp cannot be used in ListState when multiple state data is involved

[ https://issues.apache.org/jira/browse/SPARK-51587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-51587: - Issue Type: Bug (was: Task) > [PySpark] Fix an issue where timestamp cannot be used in ListStat

[jira] [Updated] (SPARK-51587) [PySpark] Fix an issue where timestamp cannot be used in ListState when multiple state data is involved

[ https://issues.apache.org/jira/browse/SPARK-51587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-51587: - Fix Version/s: 4.0.0 (was: 4.1.0) > [PySpark] Fix an issue where timestam

[jira] [Resolved] (SPARK-51252) Adding state store level metrics for last uploaded snapshot version in HDFS State Stores

[ https://issues.apache.org/jira/browse/SPARK-51252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51252. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50030 [https://gi

[jira] [Assigned] (SPARK-51252) Adding state store level metrics for last uploaded snapshot version in HDFS State Stores

[ https://issues.apache.org/jira/browse/SPARK-51252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51252: Assignee: Zeyu Chen > Adding state store level metrics for last uploaded snapshot version

[jira] [Resolved] (SPARK-51471) classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset.

[ https://issues.apache.org/jira/browse/SPARK-51471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51471. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50238 [https://gi

[jira] [Assigned] (SPARK-51471) classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset.

[ https://issues.apache.org/jira/browse/SPARK-51471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51471: Assignee: Huanli Wang > classify the ASSERT error when offset/timestamp in startOffset is

[jira] [Assigned] (SPARK-51586) Kafka continuous stream may go into infinite loop of reconfiguring

[ https://issues.apache.org/jira/browse/SPARK-51586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51586: Assignee: Vlad Rozov > Kafka continuous stream may go into infinite loop of reconfiguring

[jira] [Updated] (SPARK-51187) Implement the graceful deprecation of incorrect config introduced in SPARK-49699

[ https://issues.apache.org/jira/browse/SPARK-51187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-51187: - Fix Version/s: 3.5.5 > Implement the graceful deprecation of incorrect config introduced in > S

[jira] [Assigned] (SPARK-51097) Adding state store level metrics for last uploaded snapshot version in RocksDB

[ https://issues.apache.org/jira/browse/SPARK-51097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51097: Assignee: Zeyu Chen > Adding state store level metrics for last uploaded snapshot version

[jira] [Resolved] (SPARK-51097) Adding state store level metrics for last uploaded snapshot version in RocksDB

[ https://issues.apache.org/jira/browse/SPARK-51097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51097. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50195 [https://gi

[jira] [Resolved] (SPARK-51397) Add maintenance shutdown timeout as a configurable option

[ https://issues.apache.org/jira/browse/SPARK-51397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51397. -- Fix Version/s: 4.1.0 Assignee: Anish Shrigondekar Resolution: Fixed Issue reso

[jira] [Resolved] (SPARK-51506) Do not enforce users to implement close() in TransformWithStateInPandas

[ https://issues.apache.org/jira/browse/SPARK-51506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51506. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50272 [https://gi

[jira] [Resolved] (SPARK-51440) Classify the NPE when the topic field in kafka message is null and there is no topic option

[ https://issues.apache.org/jira/browse/SPARK-51440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51440. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50214 [https://gi

[jira] [Assigned] (SPARK-51440) Classify the NPE when the topic field in kafka message is null and there is no topic option

[ https://issues.apache.org/jira/browse/SPARK-51440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51440: Assignee: Huanli Wang > Classify the NPE when the topic field in kafka message is null an

[jira] [Assigned] (SPARK-51409) Add error classification for changelog writer related errors

[ https://issues.apache.org/jira/browse/SPARK-51409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51409: Assignee: Anish Shrigondekar > Add error classification for changelog writer related erro

[jira] [Resolved] (SPARK-51409) Add error classification for changelog writer related errors

[ https://issues.apache.org/jira/browse/SPARK-51409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51409. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50176 [https://gi

[jira] [Reopened] (SPARK-51097) Adding state store level metrics for last uploaded snapshot version in RocksDB

[ https://issues.apache.org/jira/browse/SPARK-51097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened SPARK-51097: -- Assignee: (was: Zeyu Chen) This was reverted as we observed UI being flooded with new me

[jira] [Updated] (SPARK-51097) Adding state store level metrics for last uploaded snapshot version in RocksDB

[ https://issues.apache.org/jira/browse/SPARK-51097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-51097: - Fix Version/s: (was: 4.0.0) > Adding state store level metrics for last uploaded snapshot ve

[jira] [Comment Edited] (SPARK-51097) Adding state store level metrics for last uploaded snapshot version in RocksDB

[ https://issues.apache.org/jira/browse/SPARK-51097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932810#comment-17932810 ] Jungtaek Lim edited comment on SPARK-51097 at 3/6/25 2:50 AM:

[jira] [Assigned] (SPARK-51373) Removing extra copy for column family prefix from 'ReplyChangelog'

[ https://issues.apache.org/jira/browse/SPARK-51373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51373: Assignee: Eric Marnadi > Removing extra copy for column family prefix from 'ReplyChangelo

[jira] [Resolved] (SPARK-51373) Removing extra copy for column family prefix from 'ReplyChangelog'

[ https://issues.apache.org/jira/browse/SPARK-51373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51373. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50119 [https://gi

[jira] [Assigned] (SPARK-50855) Spark Connect Support for TransformWithState In Scala

[ https://issues.apache.org/jira/browse/SPARK-50855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-50855: Assignee: Jing Zhan > Spark Connect Support for TransformWithState In Scala > ---

[jira] [Resolved] (SPARK-50855) Spark Connect Support for TransformWithState In Scala

[ https://issues.apache.org/jira/browse/SPARK-50855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-50855. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49488 [https://gi

[jira] [Resolved] (SPARK-51362) Change toJSON to use NextIterator API to reduce record-level latency

[ https://issues.apache.org/jira/browse/SPARK-51362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51362. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50124 [https://gi

[jira] [Assigned] (SPARK-51362) Change toJSON to use NextIterator API to reduce record-level latency

[ https://issues.apache.org/jira/browse/SPARK-51362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51362: Assignee: Yuchen Liu > Change toJSON to use NextIterator API to reduce record-level laten

[jira] [Resolved] (SPARK-51351) TWS PySpark implementation materializes the entire output iterator in python worker

[ https://issues.apache.org/jira/browse/SPARK-51351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51351. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50110 [https://gi

[jira] [Created] (SPARK-51351) TWS PySpark implementation materializes the entire output iterator in python worker

Jungtaek Lim created SPARK-51351: Summary: TWS PySpark implementation materializes the entire output iterator in python worker Key: SPARK-51351 URL: https://issues.apache.org/jira/browse/SPARK-51351 P

[jira] [Commented] (SPARK-51351) TWS PySpark implementation materializes the entire output iterator in python worker

[ https://issues.apache.org/jira/browse/SPARK-51351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931346#comment-17931346 ] Jungtaek Lim commented on SPARK-51351: -- Going to file a PR sooner. > TWS PySpark i

  1   2   3   4   5   6   7   8   9   10   >