[jira] [Assigned] (SPARK-50515) Add explicit read-only access API to conf in SparkContext

[ https://issues.apache.org/jira/browse/SPARK-50515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-50515: --- Assignee: Prashanth Menon > Add explicit read-only access API to conf in SparkContext > ---

[jira] [Resolved] (SPARK-50515) Add explicit read-only access API to conf in SparkContext

[ https://issues.apache.org/jira/browse/SPARK-50515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50515. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49100 [https://gith

[jira] [Updated] (SPARK-50573) Adding State Schema ID to State Rows to schema evolution

[ https://issues.apache.org/jira/browse/SPARK-50573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50573: --- Labels: pull-request-available (was: ) > Adding State Schema ID to State Rows to schema evo

[jira] [Updated] (SPARK-50360) StateStoreChangelogReader should read state store version from the changelog

[ https://issues.apache.org/jira/browse/SPARK-50360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50360: --- Labels: pull-request-available (was: ) > StateStoreChangelogReader should read state store

[jira] [Resolved] (SPARK-50565) Add transformWithState correctness test

[ https://issues.apache.org/jira/browse/SPARK-50565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-50565. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49173 [https://gi

[jira] [Assigned] (SPARK-50565) Add transformWithState correctness test

[ https://issues.apache.org/jira/browse/SPARK-50565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-50565: Assignee: Wei Liu > Add transformWithState correctness test > ---

[jira] [Assigned] (SPARK-50540) [PySpark] Fix the issue for string schema in StatefulProcessHandle

[ https://issues.apache.org/jira/browse/SPARK-50540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-50540: Assignee: Bo Gao > [PySpark] Fix the issue for string schema in StatefulProcessHandle > -

[jira] [Resolved] (SPARK-50540) [PySpark] Fix the issue for string schema in StatefulProcessHandle

[ https://issues.apache.org/jira/browse/SPARK-50540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-50540. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49138 [https://gi

[jira] [Updated] (SPARK-50578) Add support for new version of state metadata for TransformWithStateInPandas

[ https://issues.apache.org/jira/browse/SPARK-50578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhan updated SPARK-50578: -- Summary: Add support for new version of state metadata for TransformWithStateInPandas (was: Add suppo

[jira] [Created] (SPARK-50578) Add support for metadata for TransformWithStateInPandas

Jing Zhan created SPARK-50578: - Summary: Add support for metadata for TransformWithStateInPandas Key: SPARK-50578 URL: https://issues.apache.org/jira/browse/SPARK-50578 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17904288#comment-17904288 ] Asif edited comment on SPARK-33152 at 12/14/24 12:01 AM: - Latest

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/49124] in s

[jira] [Comment Edited] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824877#comment-17824877 ] Asif edited comment on SPARK-47320 at 12/13/24 11:43 PM: - Opened

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/49124] in s

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Description: The canonicalization of (buildKeys: Seq[Expression]) in the class DynamicPruningSubquery is broken

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Description: In the rule PruneFileSourcePartitions where the CatalogFileIndex gets converted to InMemoryFileInd

[jira] [Updated] (SPARK-50577) Remove trim collation checks from binary collation critical path.

[ https://issues.apache.org/jira/browse/SPARK-50577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jovan Pavlovic updated SPARK-50577: --- Summary: Remove trim collation checks from binary collation critical path. (was: Move trim

[jira] [Updated] (SPARK-50576) Upgrade `common-text` to `1.13.0`

[ https://issues.apache.org/jira/browse/SPARK-50576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50576: --- Labels: pull-request-available (was: ) > Upgrade `common-text` to `1.13.0` > --

[jira] [Created] (SPARK-50576) Upgrade `common-text` to `1.13.0`

Pan Bingkun created SPARK-50576: --- Summary: Upgrade `common-text` to `1.13.0` Key: SPARK-50576 URL: https://issues.apache.org/jira/browse/SPARK-50576 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-50575) Modifying OperatorStateMetadataV2 to enable schema evolution

Eric Marnadi created SPARK-50575: Summary: Modifying OperatorStateMetadataV2 to enable schema evolution Key: SPARK-50575 URL: https://issues.apache.org/jira/browse/SPARK-50575 Project: Spark

[jira] [Resolved] (SPARK-50546) Add subquery cast support to collation type coercion

[ https://issues.apache.org/jira/browse/SPARK-50546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-50546. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49147 [https://github.com

[jira] [Updated] (SPARK-50574) Upgrade `rexml` to 3.3.9

[ https://issues.apache.org/jira/browse/SPARK-50574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50574: --- Labels: pull-request-available (was: ) > Upgrade `rexml` to 3.3.9 > ---

[jira] [Resolved] (SPARK-50338) Make LazyTry exceptions less verbose

[ https://issues.apache.org/jira/browse/SPARK-50338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50338. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48882 [https://gith

[jira] [Resolved] (SPARK-50571) Apply Python 3.11 image in RocksDB as UI Backend daily build

[ https://issues.apache.org/jira/browse/SPARK-50571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50571. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49178 [https://

[jira] [Created] (SPARK-50573) Adding State Schema ID to State Rows to schema evolution

Eric Marnadi created SPARK-50573: Summary: Adding State Schema ID to State Rows to schema evolution Key: SPARK-50573 URL: https://issues.apache.org/jira/browse/SPARK-50573 Project: Spark Issu

[jira] [Commented] (SPARK-49872) Spark History UI -- StreamConstraintsException: String length (20054016) exceeds the maximum length (20000000)

[ https://issues.apache.org/jira/browse/SPARK-49872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905536#comment-17905536 ] Gabor Roczei commented on SPARK-49872: -- Hi [~steven.aerts], > I propose to remove

[jira] [Resolved] (SPARK-50567) Apply Python 3.11 image in No-ANSI daily build

[ https://issues.apache.org/jira/browse/SPARK-50567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50567. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49176 [https://

[jira] [Updated] (SPARK-50564) Upgrade protobuf Python package to 5.29.1

[ https://issues.apache.org/jira/browse/SPARK-50564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50564: --- Labels: pull-request-available (was: ) > Upgrade protobuf Python package to 5.29.1 > --

[jira] [Updated] (SPARK-50563) Upgrade protobuf-java to 4.29.1

[ https://issues.apache.org/jira/browse/SPARK-50563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50563: -- Parent Issue: SPARK-44111 (was: SPARK-47046) > Upgrade protobuf-java to 4.29.1 >

[jira] [Updated] (SPARK-50563) Upgrade protobuf-java to 4.29.1

[ https://issues.apache.org/jira/browse/SPARK-50563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50563: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade protobuf-java t

[jira] [Updated] (SPARK-50563) Upgrade protobuf-java to 4.29.1

[ https://issues.apache.org/jira/browse/SPARK-50563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50563: --- Labels: pull-request-available (was: ) > Upgrade protobuf-java to 4.29.1 >

[jira] [Resolved] (SPARK-50563) Upgrade protobuf-java to 4.29.1

[ https://issues.apache.org/jira/browse/SPARK-50563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50563. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49169 [https://

[jira] [Updated] (SPARK-50568) Fix KafkaMicroBatchSourceSuite to cover KafkaOffsetReaderConsumer with unit tests

[ https://issues.apache.org/jira/browse/SPARK-50568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50568: --- Labels: pull-request-available (was: ) > Fix KafkaMicroBatchSourceSuite to cover KafkaOffse

[jira] [Updated] (SPARK-50572) NullPointerException in InMemoryTableScanExec

[ https://issues.apache.org/jira/browse/SPARK-50572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50572: --- Labels: pull-request-available (was: ) > NullPointerException in InMemoryTableScanExec > --

[jira] [Created] (SPARK-50572) NullPointerException in InMemoryTableScanExec

Emil Ejbyfeldt created SPARK-50572: -- Summary: NullPointerException in InMemoryTableScanExec Key: SPARK-50572 URL: https://issues.apache.org/jira/browse/SPARK-50572 Project: Spark Issue Type:

[jira] [Updated] (SPARK-50558) Do not log all values for In and InSet if there are many

[ https://issues.apache.org/jira/browse/SPARK-50558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50558: --- Labels: pull-request-available (was: ) > Do not log all values for In and InSet if there ar

[jira] [Resolved] (SPARK-50566) Fix RocksDB code style violation

[ https://issues.apache.org/jira/browse/SPARK-50566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-50566. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49174 [https://gi

[jira] [Assigned] (SPARK-50566) Fix RocksDB code style violation

[ https://issues.apache.org/jira/browse/SPARK-50566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-50566: Assignee: Wei Liu > Fix RocksDB code style violation > >

[jira] [Commented] (SPARK-50570) ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable)

[ https://issues.apache.org/jira/browse/SPARK-50570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905473#comment-17905473 ] Sakthi commented on SPARK-50570: I would like to pick up this issue and start working on

[jira] [Updated] (SPARK-50570) ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable)

[ https://issues.apache.org/jira/browse/SPARK-50570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated SPARK-50570: --- Description: Writing data to a DynamoDB table using PySpark results in the following error: {code:java} Caus

[jira] [Updated] (SPARK-50570) ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable)

[ https://issues.apache.org/jira/browse/SPARK-50570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated SPARK-50570: --- Description: Writing data to a DynamoDB table using PySpark results in the following error: {code:java} Caus

[jira] [Created] (SPARK-50569) Clearing dataframes/views persisted in isolated Connect session

Peter Andrew created SPARK-50569: Summary: Clearing dataframes/views persisted in isolated Connect session Key: SPARK-50569 URL: https://issues.apache.org/jira/browse/SPARK-50569 Project: Spark

[jira] [Created] (SPARK-50570) ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable)

Sakthi created SPARK-50570: -- Summary: ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable) Key: SPARK-50570 URL: https://issues.apache.org/jira/browse/SPARK-50570 Proj

[jira] [Created] (SPARK-50568) Fix KafkaMicroBatchSourceSuite to cover KafkaOffsetReaderConsumer with unit tests

Dmytro Tsyliuryk created SPARK-50568: Summary: Fix KafkaMicroBatchSourceSuite to cover KafkaOffsetReaderConsumer with unit tests Key: SPARK-50568 URL: https://issues.apache.org/jira/browse/SPARK-50568

[jira] [Commented] (SPARK-49878) Spark Structured Streaming will process incorrect Kafka offsets

[ https://issues.apache.org/jira/browse/SPARK-49878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905450#comment-17905450 ] Thomas S commented on SPARK-49878: -- [~kabhwan] we already run the queries with `failOnD

[jira] [Resolved] (SPARK-50157) Using SQLConf provided by SparkSession first.

[ https://issues.apache.org/jira/browse/SPARK-50157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng resolved SPARK-50157. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48693 [https://github

[jira] [Updated] (SPARK-50567) Apply Python 3.11 image in No-ANSI daily build

[ https://issues.apache.org/jira/browse/SPARK-50567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50567: --- Labels: pull-request-available (was: ) > Apply Python 3.11 image in No-ANSI daily build > -

[jira] [Assigned] (SPARK-50566) Fix RocksDB code style violation

[ https://issues.apache.org/jira/browse/SPARK-50566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50566: -- Assignee: Apache Spark > Fix RocksDB code style violation > -

[jira] [Assigned] (SPARK-50566) Fix RocksDB code style violation

[ https://issues.apache.org/jira/browse/SPARK-50566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50566: -- Assignee: (was: Apache Spark) > Fix RocksDB code style violation > --

[jira] [Updated] (SPARK-50566) Fix RocksDB code style violation

[ https://issues.apache.org/jira/browse/SPARK-50566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-50566: - Summary: Fix RocksDB code style violation (was: RocksDB Style Improve) > Fix RocksDB code style

[jira] [Created] (SPARK-50567) Apply Python 3.11 image in No-ANSI daily build

Ruifeng Zheng created SPARK-50567: - Summary: Apply Python 3.11 image in No-ANSI daily build Key: SPARK-50567 URL: https://issues.apache.org/jira/browse/SPARK-50567 Project: Spark Issue Type: