[jira] [Updated] (SPARK-45672) Provide a unified user-facing schema for state format versions in state data source - reader

2023-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-45672: - Parent: SPARK-45511 Issue Type: Sub-task (was: Improvement) > Provide a unified

[jira] [Created] (SPARK-45672) Provide a unified user-facing schema for state format versions in state data source - reader

2023-10-25 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-45672: Summary: Provide a unified user-facing schema for state format versions in state data source - reader Key: SPARK-45672 URL: https://issues.apache.org/jira/browse/SPARK-45672

[jira] [Created] (SPARK-45671) Implement an option similar to corrupt record column in State Data Source Reader

2023-10-25 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-45671: Summary: Implement an option similar to corrupt record column in State Data Source Reader Key: SPARK-45671 URL: https://issues.apache.org/jira/browse/SPARK-45671

[jira] [Commented] (SPARK-38723) Test the error class: CONCURRENT_QUERY

2023-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779748#comment-17779748 ] Jungtaek Lim commented on SPARK-38723: -- (I just asked about the Jira ccount to the PR author. I'll

[jira] [Resolved] (SPARK-38723) Test the error class: CONCURRENT_QUERY

2023-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-38723. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43405

[jira] [Resolved] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45663. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43527

[jira] [Assigned] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45663: Assignee: Yang Jie > Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft` >

[jira] [Assigned] (SPARK-44407) Clean up the compilation warnings related to `it will become a keyword in Scala 3`

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44407: Assignee: Yang Jie > Clean up the compilation warnings related to `it will become a keyword in

[jira] [Resolved] (SPARK-44407) Clean up the compilation warnings related to `it will become a keyword in Scala 3`

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44407. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43529

[jira] [Updated] (SPARK-45670) SparkSubmit does not support --total-executor-cores when deploying on K8s

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45670: --- Labels: pull-request-available (was: ) > SparkSubmit does not support

[jira] [Created] (SPARK-45670) SparkSubmit does not support --total-executor-cores when deploying on K8s

2023-10-25 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-45670: - Summary: SparkSubmit does not support --total-executor-cores when deploying on K8s Key: SPARK-45670 URL: https://issues.apache.org/jira/browse/SPARK-45670 Project: Spark

[jira] [Updated] (SPARK-43923) [CONNECT] Post listenerBus events during ExecutePlanRequest

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43923: --- Labels: pull-request-available (was: ) > [CONNECT] Post listenerBus events during

[jira] [Updated] (SPARK-45669) Ensure the continuity of rolling log index

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45669: --- Labels: pull-request-available (was: ) > Ensure the continuity of rolling log index >

[jira] [Updated] (SPARK-45667) Clean up the deprecated API usage related to `IterableOnceExtensionMethods`.

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45667: --- Labels: pull-request-available (was: ) > Clean up the deprecated API usage related to

[jira] [Updated] (SPARK-45668) Improve the assert message in `RollingEventLogFilesFileReader`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45668: --- Labels: pull-request-available (was: ) > Improve the assert message in

[jira] [Resolved] (SPARK-45665) Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other branches

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45665. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43496

[jira] [Assigned] (SPARK-45665) Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other branches

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45665: Assignee: Yang Jie > Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other >

[jira] [Created] (SPARK-45669) Ensure the continuity of rolling log index

2023-10-25 Thread shuyouZZ (Jira)
shuyouZZ created SPARK-45669: Summary: Ensure the continuity of rolling log index Key: SPARK-45669 URL: https://issues.apache.org/jira/browse/SPARK-45669 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-45668) Improve the assert message in `RollingEventLogFilesFileReader`

2023-10-25 Thread shuyouZZ (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shuyouZZ updated SPARK-45668: - Description: Currently the assert message in `RollingEventLogFilesFileReader` is not clear. When the

[jira] [Created] (SPARK-45668) Improve the assert message in `RollingEventLogFilesFileReader`

2023-10-25 Thread shuyouZZ (Jira)
shuyouZZ created SPARK-45668: Summary: Improve the assert message in `RollingEventLogFilesFileReader` Key: SPARK-45668 URL: https://issues.apache.org/jira/browse/SPARK-45668 Project: Spark

[jira] [Resolved] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45650. -- Resolution: Not A Problem > fix dev/mina get scala 2.12 > > >

[jira] [Commented] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779724#comment-17779724 ] Yang Jie commented on SPARK-45650: -- {code:java} [error] spark-sql-api: Failed binary compatibility

[jira] [Created] (SPARK-45667) Clean up the deprecated API usage related to `IterableOnceExtensionMethods`.

2023-10-25 Thread Yang Jie (Jira)
Yang Jie created SPARK-45667: Summary: Clean up the deprecated API usage related to `IterableOnceExtensionMethods`. Key: SPARK-45667 URL: https://issues.apache.org/jira/browse/SPARK-45667 Project: Spark

[jira] [Updated] (SPARK-45665) Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other branches

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45665: --- Labels: pull-request-available (was: ) > Uses different ORACLE_DOCKER_IMAGE_NAME in the

[jira] [Assigned] (SPARK-45635) Cleanup unused import for PySpark testing

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45635: Assignee: Haejoon Lee > Cleanup unused import for PySpark testing >

[jira] [Resolved] (SPARK-45635) Cleanup unused import for PySpark testing

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45635. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43489

[jira] [Resolved] (SPARK-45634) Remove `get_dtype_counts` from Pandas API on Spark

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45634. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43488

[jira] [Updated] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-19335: --- Labels: pull-request-available (was: ) > Spark should support doing an efficient DataFrame

[jira] [Assigned] (SPARK-45634) Remove `get_dtype_counts` from Pandas API on Spark

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45634: Assignee: Haejoon Lee > Remove `get_dtype_counts` from Pandas API on Spark >

[jira] [Assigned] (SPARK-45136) Improve ClosureCleaner to support closures defined in Ammonite REPL

2023-10-25 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-45136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-45136: - Assignee: Vsevolod Stepanov (was: Herman van Hövell) > Improve ClosureCleaner

[jira] [Resolved] (SPARK-45136) Improve ClosureCleaner to support closures defined in Ammonite REPL

2023-10-25 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-45136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-45136. --- Fix Version/s: 4.0.0 Resolution: Fixed > Improve ClosureCleaner to support

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-45658: Target Version/s: (was: 3.5.1) > Canonicalization of DynamicPruningSubquery is broken >

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-45658: Affects Version/s: (was: 3.5.1) > Canonicalization of DynamicPruningSubquery is broken >

[jira] [Assigned] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45661: Assignee: Hyukjin Kwon > Add toNullable in StructType, MapType and ArrayType >

[jira] [Resolved] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45661. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43523

[jira] [Updated] (SPARK-44440) Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-0: --- Labels: pull-request-available (was: ) > Use thread pool to perform maintenance activity

[jira] [Updated] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44124: -- Parent: (was: SPARK-44111) Issue Type: Improvement (was: Sub-task) > Upgrade AWS

[jira] [Updated] (SPARK-45652) SPJ: Handle empty input partitions after dynamic filtering

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45652: --- Labels: pull-request-available (was: ) > SPJ: Handle empty input partitions after dynamic

[jira] [Updated] (SPARK-29615) Add insertInto method with byName parameter in DataFrameWriter

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-29615: --- Labels: pull-request-available (was: ) > Add insertInto method with byName parameter in

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Created] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
dinesh sachdev created SPARK-45666: -- Summary: spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set. Key: SPARK-45666 URL: https://issues.apache.org/jira/browse/SPARK-45666

[jira] [Updated] (SPARK-45666) spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.

2023-10-25 Thread dinesh sachdev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh sachdev updated SPARK-45666: --- Description: We had a requirement to write Custom

[jira] [Resolved] (SPARK-45660) Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule

2023-10-25 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45660. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43524

[jira] [Assigned] (SPARK-45660) Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule

2023-10-25 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-45660: Assignee: Jan-Ole Sasse > Re-use Literal objects when replacing timestamps in the

[jira] [Updated] (SPARK-45545) Pass SSLOptions wherever we create a SparkTransportConf

2023-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45545: -- Summary: Pass SSLOptions wherever we create a SparkTransportConf (was: [CORE] Pass

[jira] [Updated] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level

2023-10-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-40154: - Priority: Trivial (was: Minor) > PySpark: DataFrame.cache docstring gives wrong storage level

[jira] [Resolved] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level

2023-10-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40154. -- Fix Version/s: 3.5.1 4.0.0 3.4.2 Resolution:

[jira] [Assigned] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level

2023-10-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-40154: Assignee: Paul Staab > PySpark: DataFrame.cache docstring gives wrong storage level >

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779460#comment-17779460 ] Steve Loughran commented on SPARK-44124: good document # I think you could consider cutting the

[jira] [Assigned] (SPARK-45209) Flame Graph Support For Executor Thread Dump Page

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45209: Assignee: Kent Yao > Flame Graph Support For Executor Thread Dump Page >

[jira] [Resolved] (SPARK-45209) Flame Graph Support For Executor Thread Dump Page

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45209. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42988

[jira] [Created] (SPARK-45665) Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other branches

2023-10-25 Thread Yang Jie (Jira)
Yang Jie created SPARK-45665: Summary: Uses different ORACLE_DOCKER_IMAGE_NAME in the scheduled builds in other branches Key: SPARK-45665 URL: https://issues.apache.org/jira/browse/SPARK-45665 Project:

[jira] [Commented] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-25 Thread tangjiafu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779431#comment-17779431 ] tangjiafu commented on SPARK-45650: --- in this PR ( https://github.com/apache/spark/pull/43456 )I meet

[jira] [Updated] (SPARK-45664) Introduce a mapper for orc compression codecs

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45664: --- Labels: pull-request-available (was: ) > Introduce a mapper for orc compression codecs >

[jira] [Updated] (SPARK-45664) Introduce a mapper for orc compression codecs

2023-10-25 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45664: --- Description: Currently, Spark supported all the orc compression codecs, but the orc supported

[jira] [Updated] (SPARK-45664) Introduce a mapper for orc compression codecs

2023-10-25 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45664: --- Description: Currently, Spark supported all the orc compression codecs, but the orc supported

[jira] [Assigned] (SPARK-45481) Introduce a mapper for parquet compression codecs

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45481: -- Assignee: Apache Spark (was: Jiaan Geng) > Introduce a mapper for parquet

[jira] [Assigned] (SPARK-45481) Introduce a mapper for parquet compression codecs

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45481: -- Assignee: Jiaan Geng (was: Apache Spark) > Introduce a mapper for parquet

[jira] [Updated] (SPARK-45481) Introduce a mapper for parquet compression codecs

2023-10-25 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45481: --- Description: Currently, Spark supported all the parquet compression codecs, but the parquet

[jira] [Updated] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45663: --- Labels: pull-request-available (was: ) > Replace `IterableOnceOps#aggregate` with

[jira] [Assigned] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45663: -- Assignee: (was: Apache Spark) > Replace `IterableOnceOps#aggregate` with

[jira] [Assigned] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45663: -- Assignee: Apache Spark > Replace `IterableOnceOps#aggregate` with

[jira] [Assigned] (SPARK-45629) Fix `Implicit definition should have explicit type`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45629: -- Assignee: (was: Apache Spark) > Fix `Implicit definition should have explicit

[jira] [Assigned] (SPARK-45629) Fix `Implicit definition should have explicit type`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45629: -- Assignee: Apache Spark > Fix `Implicit definition should have explicit type` >

[jira] [Assigned] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45661: -- Assignee: Apache Spark > Add toNullable in StructType, MapType and ArrayType >

[jira] [Assigned] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45661: -- Assignee: (was: Apache Spark) > Add toNullable in StructType, MapType and

[jira] [Updated] (SPARK-45629) Fix `Implicit definition should have explicit type`

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45629: --- Labels: pull-request-available (was: ) > Fix `Implicit definition should have explicit

[jira] [Assigned] (SPARK-45606) Release restrictions on multi-layer runtime filter

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45606: -- Assignee: Jiaan Geng (was: Apache Spark) > Release restrictions on multi-layer

[jira] [Assigned] (SPARK-45606) Release restrictions on multi-layer runtime filter

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45606: -- Assignee: Apache Spark (was: Jiaan Geng) > Release restrictions on multi-layer

[jira] [Assigned] (SPARK-45606) Release restrictions on multi-layer runtime filter

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45606: -- Assignee: Jiaan Geng (was: Apache Spark) > Release restrictions on multi-layer

[jira] [Assigned] (SPARK-45606) Release restrictions on multi-layer runtime filter

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45606: -- Assignee: Apache Spark (was: Jiaan Geng) > Release restrictions on multi-layer

[jira] [Assigned] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives (.har)

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-39910: -- Assignee: (was: Apache Spark) > DataFrameReader API cannot read files from

[jira] [Assigned] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives (.har)

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-39910: -- Assignee: Apache Spark > DataFrameReader API cannot read files from hadoop archives

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779407#comment-17779407 ] Lantao Jin commented on SPARK-44124: Is it possible to convert this JIRA to an issue for adding more

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779405#comment-17779405 ] Lantao Jin commented on SPARK-44124: Added a design doc (more like a plan) in description. >

[jira] [Updated] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-44124: --- Description: Here is a design doc:

[jira] [Assigned] (SPARK-45656) Fix observation when named observations with the same name on different datasets.

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45656: Assignee: Takuya Ueshin > Fix observation when named observations with the same name on

[jira] [Resolved] (SPARK-45656) Fix observation when named observations with the same name on different datasets.

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45656. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43519

[jira] [Commented] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779397#comment-17779397 ] Hyukjin Kwon commented on SPARK-45662: -- Can you provide a reproducer please > Pyspark union

[jira] [Created] (SPARK-45663) Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`

2023-10-25 Thread Yang Jie (Jira)
Yang Jie created SPARK-45663: Summary: Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft` Key: SPARK-45663 URL: https://issues.apache.org/jira/browse/SPARK-45663 Project: Spark

[jira] [Commented] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-25 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779383#comment-17779383 ] Yang Jie commented on SPARK-45650: -- Do you have a more detailed error stack? `[info] [launcher] getting

[jira] [Updated] (SPARK-45660) Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45660: --- Labels: pull-request-available (was: ) > Re-use Literal objects when replacing timestamps

[jira] [Commented] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779370#comment-17779370 ] Yi Zhu commented on SPARK-45662: ping [~gurwls223] Could you take a look? I have checked that the env is

[jira] [Updated] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhu updated SPARK-45662: --- Description: !image-2023-10-25-15-01-28-896.png|width=1795,height=194!     The code is  {code:java}    

[jira] [Updated] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhu updated SPARK-45662: --- Attachment: image-2023-10-25-15-01-28-896.png > Pyspark union method throw

[jira] [Updated] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhu updated SPARK-45662: --- Description: !image-2023-10-25-15-01-28-896.png|width=1795,height=194! > Pyspark union method throw

[jira] [Created] (SPARK-45662) Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'

2023-10-25 Thread Yi Zhu (Jira)
Yi Zhu created SPARK-45662: -- Summary: Pyspark union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed' Key: SPARK-45662 URL: https://issues.apache.org/jira/browse/SPARK-45662

[jira] [Updated] (SPARK-45640) Fix flaky ProtobufCatalystDataConversionSuite

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45640: - Fix Version/s: 3.4.2 3.5.1 > Fix flaky ProtobufCatalystDataConversionSuite >

[jira] [Updated] (SPARK-45588) Minor scaladoc improvement in StreamingForeachBatchHelper

2023-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45588: - Fix Version/s: 3.4.2 3.5.1 > Minor scaladoc improvement in

[jira] [Updated] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45661: --- Labels: pull-request-available (was: ) > Add toNullable in StructType, MapType and

[jira] [Created] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-25 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-45661: Summary: Add toNullable in StructType, MapType and ArrayType Key: SPARK-45661 URL: https://issues.apache.org/jira/browse/SPARK-45661 Project: Spark Issue

[jira] [Commented] (SPARK-43108) org.apache.spark.storage.StorageStatus NotSerializableException when try to access StorageStatus in a MapPartitionsFunction

2023-10-25 Thread Berg Lloyd-Haig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779355#comment-17779355 ] Berg Lloyd-Haig commented on SPARK-43108: - We are hitting this when reading from Snowflake using

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-25 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: I do not really know if this is a bug, but I am at the end with my knowledge. A Spark

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-25 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Issue Type: Bug (was: Question) > After upgrading to Spark 3.4.1 and 3.5.0 we receive

  1   2   >