[jira] [Commented] (SPARK-43338) Support modify the SESSION_CATALOG_NAME value

[ https://issues.apache.org/jira/browse/SPARK-43338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720788#comment-17720788 ] Kent Yao commented on SPARK-43338: -- Support multiple Hive metastore servers is a big feature, which can

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Commented] (SPARK-43413) IN subquery ListQuery has wrong nullability

[ https://issues.apache.org/jira/browse/SPARK-43413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720780#comment-17720780 ] ci-cassandra.apache.org commented on SPARK-43413: - User 'jchen5' has created a pull

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Created] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

Fei Wang created SPARK-43419: Summary: [K8S] Make limit.cores be able to be fallen back to request.cores Key: SPARK-43419 URL: https://issues.apache.org/jira/browse/SPARK-43419 Project: Spark

[jira] [Commented] (SPARK-43014) spark.app.submitTime is not right in k8s cluster mode

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720770#comment-17720770 ] Snoot.io commented on SPARK-43014: -- User 'zhouyifan279' has created a pull request for this issue:

[jira] [Commented] (SPARK-42668) Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort

[ https://issues.apache.org/jira/browse/SPARK-42668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720769#comment-17720769 ] Snoot.io commented on SPARK-42668: -- User 'HyukjinKwon' has created a pull request for this issue:

[jira] [Assigned] (SPARK-43343) Spark Streaming is not able to read a .txt file whose name has [] special character

[ https://issues.apache.org/jira/browse/SPARK-43343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-43343: Assignee: Siying Dong > Spark Streaming is not able to read a .txt file whose name has

[jira] [Resolved] (SPARK-43343) Spark Streaming is not able to read a .txt file whose name has [] special character

[ https://issues.apache.org/jira/browse/SPARK-43343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-43343. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41022

[jira] [Commented] (SPARK-43357) Spark AWS Glue date partition push down broken

[ https://issues.apache.org/jira/browse/SPARK-43357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720763#comment-17720763 ] Snoot.io commented on SPARK-43357: -- User 'stijndehaes' has created a pull request for this issue:

[jira] [Resolved] (SPARK-43296) Migrate Spark Connect session errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43296. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40964

[jira] [Assigned] (SPARK-43296) Migrate Spark Connect session errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43296: - Assignee: Haejoon Lee > Migrate Spark Connect session errors into error class >

[jira] [Updated] (SPARK-42514) Scala Client add partition transforms functions

[ https://issues.apache.org/jira/browse/SPARK-42514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-42514: - Epic Link: SPARK-42554 > Scala Client add partition transforms functions >

[jira] [Created] (SPARK-43418) add SparkSession.Builder.getOrCreate()

Herman van Hövell created SPARK-43418: - Summary: add SparkSession.Builder.getOrCreate() Key: SPARK-43418 URL: https://issues.apache.org/jira/browse/SPARK-43418 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-43401) Upgrade buf to v1.18.0

[ https://issues.apache.org/jira/browse/SPARK-43401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43401: - Assignee: BingKun Pan > Upgrade buf to v1.18.0 > -- > >

[jira] [Resolved] (SPARK-43401) Upgrade buf to v1.18.0

[ https://issues.apache.org/jira/browse/SPARK-43401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43401. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41087

[jira] [Resolved] (SPARK-43397) Log executor decommission duration

[ https://issues.apache.org/jira/browse/SPARK-43397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43397. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41077

[jira] [Updated] (SPARK-43397) Log executor decommission duration in executorLost method

[ https://issues.apache.org/jira/browse/SPARK-43397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43397: -- Summary: Log executor decommission duration in executorLost method (was: Log executor

[jira] [Updated] (SPARK-43397) Log executor decommission duration in executorLost method

[ https://issues.apache.org/jira/browse/SPARK-43397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43397: -- Affects Version/s: 3.5.0 (was: 3.4.0) > Log executor decommission

[jira] [Assigned] (SPARK-43397) Log executor decommission duration

[ https://issues.apache.org/jira/browse/SPARK-43397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43397: - Assignee: Zhongwei Zhu > Log executor decommission duration >

[jira] [Created] (SPARK-43417) Improve CBO stats

Huaxin Gao created SPARK-43417: -- Summary: Improve CBO stats Key: SPARK-43417 URL: https://issues.apache.org/jira/browse/SPARK-43417 Project: Spark Issue Type: New Feature Components:

[jira] [Assigned] (SPARK-43404) Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch

[ https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-43404: Assignee: Anish Shrigondekar > Filter current version while reusing sst files for

[jira] [Resolved] (SPARK-43404) Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch

[ https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-43404. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41089

[jira] [Created] (SPARK-43416) Fix the bug where the ProduceEncoder#tuples fields names are different from server

Zhen Li created SPARK-43416: --- Summary: Fix the bug where the ProduceEncoder#tuples fields names are different from server Key: SPARK-43416 URL: https://issues.apache.org/jira/browse/SPARK-43416 Project:

[jira] [Updated] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

[ https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43398: -- Affects Version/s: 3.4.0 3.3.2 3.2.4

[jira] [Resolved] (SPARK-43414) Fix flakiness in Kafka RDD suites due to port binding configuration issue

[ https://issues.apache.org/jira/browse/SPARK-43414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43414. --- Fix Version/s: 3.4.1 Resolution: Fixed Issue resolved by pull request 41095

[jira] [Created] (SPARK-43415) Impl mapValues for KVGDS#mapValues

Zhen Li created SPARK-43415: --- Summary: Impl mapValues for KVGDS#mapValues Key: SPARK-43415 URL: https://issues.apache.org/jira/browse/SPARK-43415 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-43413) IN subquery ListQuery has wrong nullability

[ https://issues.apache.org/jira/browse/SPARK-43413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-43413: -- Description: IN subquery expressions are incorrectly marked as non-nullable, even when they are

[jira] [Updated] (SPARK-43413) IN subquery ListQuery has wrong nullability

[ https://issues.apache.org/jira/browse/SPARK-43413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-43413: -- Description: IN subquery expressions are incorrectly always marked as non-nullable, even when they

[jira] [Created] (SPARK-43414) Fix flakiness in Kafka RDD suites due to port binding configuration issue

Josh Rosen created SPARK-43414: -- Summary: Fix flakiness in Kafka RDD suites due to port binding configuration issue Key: SPARK-43414 URL: https://issues.apache.org/jira/browse/SPARK-43414 Project: Spark

[jira] [Updated] (SPARK-43413) IN subquery ListQuery has wrong nullability

[ https://issues.apache.org/jira/browse/SPARK-43413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-43413: -- Description: IN subquery expressions currently are marked as nullable if and only if the

[jira] [Created] (SPARK-43413) IN subquery ListQuery has wrong nullability

Jack Chen created SPARK-43413: - Summary: IN subquery ListQuery has wrong nullability Key: SPARK-43413 URL: https://issues.apache.org/jira/browse/SPARK-43413 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-43411) Can't union dataframes with # in subcolumn name

[ https://issues.apache.org/jira/browse/SPARK-43411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rudhra Raveendran resolved SPARK-43411. --- Resolution: Won't Fix Turns out this issue isn't present in later versions of Spark

[jira] [Created] (SPARK-43412) Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs

Xinrong Meng created SPARK-43412: Summary: Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs Key: SPARK-43412 URL: https://issues.apache.org/jira/browse/SPARK-43412 Project:

[jira] [Commented] (SPARK-38471) Use error classes in org.apache.spark.rdd

[ https://issues.apache.org/jira/browse/SPARK-38471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720631#comment-17720631 ] xin chen commented on SPARK-38471: -- Hey [~maxgekk], I'm currently working on this ticket. Could you

[jira] [Commented] (SPARK-38471) Use error classes in org.apache.spark.rdd

[ https://issues.apache.org/jira/browse/SPARK-38471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720628#comment-17720628 ] xin chen commented on SPARK-38471: -- awesome, thanks.  > Use error classes in org.apache.spark.rdd >

[jira] [Created] (SPARK-43411) Can't union dataframes with # in subcolumn name

Rudhra Raveendran created SPARK-43411: - Summary: Can't union dataframes with # in subcolumn name Key: SPARK-43411 URL: https://issues.apache.org/jira/browse/SPARK-43411 Project: Spark

[jira] [Assigned] (SPARK-43410) Improve vectorized loop for Packed skipValues

[ https://issues.apache.org/jira/browse/SPARK-43410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-43410: Assignee: xiaochen zhou > Improve vectorized loop for Packed skipValues >

[jira] [Resolved] (SPARK-43410) Improve vectorized loop for Packed skipValues

[ https://issues.apache.org/jira/browse/SPARK-43410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-43410. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41092

[jira] [Updated] (SPARK-43410) Improve vectorized loop for Packed skipValues

[ https://issues.apache.org/jira/browse/SPARK-43410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaochen zhou updated SPARK-43410: -- Description: Improve vectorized loop for Packed skipValues (was: Improve vectorized loop for

[jira] [Commented] (SPARK-43389) spark.read.csv throws NullPointerException when lineSep is set to None

[ https://issues.apache.org/jira/browse/SPARK-43389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720568#comment-17720568 ] Zach Liu commented on SPARK-43389: -- that's why i set the type as "Improvement" and the priority as

[jira] [Resolved] (SPARK-43292) ArtifactManagerSuite can't run using maven

[ https://issues.apache.org/jira/browse/SPARK-43292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43292. --- Fix Version/s: 3.5.0 Assignee: Yang Jie Resolution: Fixed >

[jira] [Commented] (SPARK-43410) Improve vectorized loop for Packed skipValues

[ https://issues.apache.org/jira/browse/SPARK-43410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720525#comment-17720525 ] xiaochen zhou commented on SPARK-43410: --- https://github.com/apache/spark/pull/41092 > Improve

[jira] [Created] (SPARK-43410) Improve vectorized loop for Packed skipValues

xiaochen zhou created SPARK-43410: - Summary: Improve vectorized loop for Packed skipValues Key: SPARK-43410 URL: https://issues.apache.org/jira/browse/SPARK-43410 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-41532) DF operations that involve multiple data frames should fail if sessions don't match

[ https://issues.apache.org/jira/browse/SPARK-41532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-41532. --- Fix Version/s: 3.5.0 Resolution: Fixed > DF operations that involve multiple

[jira] [Assigned] (SPARK-41532) DF operations that involve multiple data frames should fail if sessions don't match

[ https://issues.apache.org/jira/browse/SPARK-41532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-41532: - Assignee: Jia Fan > DF operations that involve multiple data frames should

[jira] [Created] (SPARK-43409) Add documentation for protobuf deserialization options

Parth Upadhyay created SPARK-43409: -- Summary: Add documentation for protobuf deserialization options Key: SPARK-43409 URL: https://issues.apache.org/jira/browse/SPARK-43409 Project: Spark

[jira] [Updated] (SPARK-39281) Speed up Timestamp type inference of legacy format in JSON/CSV data source

[ https://issues.apache.org/jira/browse/SPARK-39281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-39281: Summary: Speed up Timestamp type inference of legacy format in JSON/CSV data source (was: Fasten

[jira] [Updated] (SPARK-39281) Speed up Timestamp type inference of legacy format in JSON/CSV data source

[ https://issues.apache.org/jira/browse/SPARK-39281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-39281: Description: The optimization of {{DefaultTimestampFormatter}} has been implemented in 

[jira] [Updated] (SPARK-43408) Spark caching in the context of a single job

[ https://issues.apache.org/jira/browse/SPARK-43408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-43408: --- Description: Does caching benefit a spark job with only a single action in it? Spark IIRC already

[jira] [Updated] (SPARK-43408) Spark caching in the context of a single job

[ https://issues.apache.org/jira/browse/SPARK-43408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-43408: --- Description: Does caching benefit a spark job with only a single action in it? Spark IIRC already

[jira] [Updated] (SPARK-43408) Spark caching in the context of a single job

[ https://issues.apache.org/jira/browse/SPARK-43408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-43408: --- Description: Does caching benefit a spark job with only a single action in it? Spark IIRC already

[jira] [Updated] (SPARK-42551) Support more subexpression elimination cases

[ https://issues.apache.org/jira/browse/SPARK-42551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-42551: Description: h1. *Design Sketch* h2. How to support more subexpressions elimination cases * Get all

[jira] [Updated] (SPARK-42551) Support more subexpression elimination cases

[ https://issues.apache.org/jira/browse/SPARK-42551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-42551: Description: h1. *Design Sketch* h2. How to support more subexpressions elimination cases * Get all

[jira] [Updated] (SPARK-42551) Support more subexpression elimination cases

[ https://issues.apache.org/jira/browse/SPARK-42551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-42551: Description: h1. *Design Sketch* h2. How to support more subexpressions elimination cases * Get all

[jira] [Commented] (SPARK-43406) enable spark sql to drop multiple partitions in one call

[ https://issues.apache.org/jira/browse/SPARK-43406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720446#comment-17720446 ] ASF GitHub Bot commented on SPARK-43406: User 'chenruotao' has created a pull request for this

[jira] [Created] (SPARK-43408) Spark caching in the context of a single job

Faiz Halde created SPARK-43408: -- Summary: Spark caching in the context of a single job Key: SPARK-43408 URL: https://issues.apache.org/jira/browse/SPARK-43408 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43400) create table support the PRIMARY KEY keyword

[ https://issues.apache.org/jira/browse/SPARK-43400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43400: -- Description: apache paimon and hudi support primary key definitions. It is necessary to support the primary

[jira] [Created] (SPARK-43407) Can executors recover/reuse shuffle files upon failure?

Faiz Halde created SPARK-43407: -- Summary: Can executors recover/reuse shuffle files upon failure? Key: SPARK-43407 URL: https://issues.apache.org/jira/browse/SPARK-43407 Project: Spark Issue

[jira] [Updated] (SPARK-43400) create table support the PRIMARY KEY keyword

[ https://issues.apache.org/jira/browse/SPARK-43400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43400: -- Description: apache paimon and hudi support primary key definitions. It is necessary to support the primary

[jira] [Updated] (SPARK-43406) enable spark sql to drop multiple partitions in one call

[ https://issues.apache.org/jira/browse/SPARK-43406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated SPARK-43406: --- Description: Now spark sql cannot drop multiple partitions in one call, so I fix it With this

[jira] [Created] (SPARK-43406) enable spark sql to drop multiple partitions in one call

chenruotao created SPARK-43406: -- Summary: enable spark sql to drop multiple partitions in one call Key: SPARK-43406 URL: https://issues.apache.org/jira/browse/SPARK-43406 Project: Spark Issue

[jira] [Created] (SPARK-43405) Remove useless code in `ScriptInputOutputSchema`

Jia Fan created SPARK-43405: --- Summary: Remove useless code in `ScriptInputOutputSchema` Key: SPARK-43405 URL: https://issues.apache.org/jira/browse/SPARK-43405 Project: Spark Issue Type:

[jira] [Commented] (SPARK-43404) Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch

[ https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720390#comment-17720390 ] Anish Shrigondekar commented on SPARK-43404: PR here: