[jira] [Commented] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598183#comment-17598183 ] Apache Spark commented on SPARK-40271: -- User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598182#comment-17598182 ] Apache Spark commented on SPARK-40271: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39896) The structural integrity of the plan is broken after UnwrapCastInBinaryComparison

2022-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39896: --- Assignee: Fu Chen > The structural integrity of the plan is broken after >

[jira] [Resolved] (SPARK-39896) The structural integrity of the plan is broken after UnwrapCastInBinaryComparison

2022-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39896. - Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue resolved by pull

[jira] [Created] (SPARK-40285) Simplify the roundTo[Numeric] for Decimal

2022-08-30 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-40285: -- Summary: Simplify the roundTo[Numeric] for Decimal Key: SPARK-40285 URL: https://issues.apache.org/jira/browse/SPARK-40285 Project: Spark Issue Type:

[jira] [Updated] (SPARK-40284) spark concurrent overwrite mode writes data to files in HDFS format, all request data write success

2022-08-30 Thread Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu updated SPARK-40284: Description: We use Spark as a service. The same Spark service needs to handle multiple requests, but I have a

[jira] [Created] (SPARK-40284) spark concurrent overwrite mode writes data to files in HDFS format, all request data write success

2022-08-30 Thread Liu (Jira)
Liu created SPARK-40284: --- Summary: spark concurrent overwrite mode writes data to files in HDFS format, all request data write success Key: SPARK-40284 URL: https://issues.apache.org/jira/browse/SPARK-40284

[jira] [Comment Edited] (SPARK-33598) Support Java Class with circular references

2022-08-30 Thread Santokh Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598165#comment-17598165 ] Santokh Singh edited comment on SPARK-33598 at 8/31/22 4:38 AM: *Facing

[jira] [Commented] (SPARK-33598) Support Java Class with circular references

2022-08-30 Thread Santokh Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598165#comment-17598165 ] Santokh Singh commented on SPARK-33598: --- *Facing same exception, Spark Version 3.2.2* *Using avro

[jira] [Comment Edited] (SPARK-33598) Support Java Class with circular references

2022-08-30 Thread Santokh Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598165#comment-17598165 ] Santokh Singh edited comment on SPARK-33598 at 8/31/22 4:27 AM: *Facing

[jira] [Updated] (SPARK-39971) ANALYZE TABLE makes some queries run forever

2022-08-30 Thread Felipe (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felipe updated SPARK-39971: --- Attachment: explainMode-cost.zip > ANALYZE TABLE makes some queries run forever >

[jira] [Updated] (SPARK-39971) ANALYZE TABLE makes some queries run forever

2022-08-30 Thread Felipe (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felipe updated SPARK-39971: --- Attachment: (was: explainMode-cost.zip) > ANALYZE TABLE makes some queries run forever >

[jira] [Reopened] (SPARK-39971) ANALYZE TABLE makes some queries run forever

2022-08-30 Thread Felipe (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felipe reopened SPARK-39971: Sorry. I found my change was causing the issue in some of TPC-DS, but not all. For the query 24 specifically

[jira] [Resolved] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40271. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37722

[jira] [Assigned] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40271: - Assignee: Haejoon Lee > Support list type for pyspark.sql.functions.lit >

[jira] [Commented] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598138#comment-17598138 ] Hyukjin Kwon commented on SPARK-40274: -- The error is likely from other libraries assuming from the

[jira] [Resolved] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40274. -- Resolution: Invalid > ArrayIndexOutOfBoundsException in BytecodeReadingParanamer >

[jira] [Commented] (SPARK-40282) DataType argument in StructType.add is incorrectly throwing scala.MatchError

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598132#comment-17598132 ] Hyukjin Kwon commented on SPARK-40282: -- We don't have this problem in the languages supported by

[jira] [Updated] (SPARK-40282) DataType argument in StructType.add is incorrectly throwing scala.MatchError

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40282: - Component/s: SQL (was: Spark Core) > DataType argument in StructType.add

[jira] [Updated] (SPARK-40282) DataType argument in StructType.add is incorrectly throwing scala.MatchError

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40282: - Priority: Major (was: Blocker) > DataType argument in StructType.add is incorrectly throwing

[jira] [Created] (SPARK-40283) Update mima's previousSparkVersion to 3.3.0

2022-08-30 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-40283: --- Summary: Update mima's previousSparkVersion to 3.3.0 Key: SPARK-40283 URL: https://issues.apache.org/jira/browse/SPARK-40283 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-39971) ANALYZE TABLE makes some queries run forever

2022-08-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-39971. - Resolution: Not A Problem > ANALYZE TABLE makes some queries run forever >

[jira] [Commented] (SPARK-39971) ANALYZE TABLE makes some queries run forever

2022-08-30 Thread Felipe (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598121#comment-17598121 ] Felipe commented on SPARK-39971: I found the issue was caused by a customization in my code. We are

[jira] [Commented] (SPARK-31001) Add ability to create a partitioned table via catalog.createTable()

2022-08-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598115#comment-17598115 ] Nicholas Chammas commented on SPARK-31001: -- What's {{{}__partition_columns{}}}? Is that

[jira] [Updated] (SPARK-40282) DataType argument in StructType.add is incorrectly throwing scala.MatchError

2022-08-30 Thread M. Manna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M. Manna updated SPARK-40282: - Summary: DataType argument in StructType.add is incorrectly throwing scala.MatchError (was:

[jira] [Updated] (SPARK-40282) DataType argument in StructType.add is incorrectly throwing scala.MatchError

2022-08-30 Thread M. Manna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M. Manna updated SPARK-40282: - Description: *Problem Description* as part of contract mentioned here, Spark should be able to support

[jira] [Updated] (SPARK-40282) IntegerType is missed in "ExternalDataTypeForInput" function

2022-08-30 Thread M. Manna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M. Manna updated SPARK-40282: - Attachment: SparkApplication.kt > IntegerType is missed in "ExternalDataTypeForInput" function >

[jira] [Created] (SPARK-40282) IntegerType is missed in "ExternalDataTypeForInput" function

2022-08-30 Thread M. Manna (Jira)
M. Manna created SPARK-40282: Summary: IntegerType is missed in "ExternalDataTypeForInput" function Key: SPARK-40282 URL: https://issues.apache.org/jira/browse/SPARK-40282 Project: Spark Issue

[jira] [Updated] (SPARK-40282) IntegerType is missed in "ExternalDataTypeForInput" function

2022-08-30 Thread M. Manna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M. Manna updated SPARK-40282: - Attachment: retailstore.csv > IntegerType is missed in "ExternalDataTypeForInput" function >

[jira] [Assigned] (SPARK-40281) Memory Profiler on Executors

2022-08-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-40281: Assignee: (was: Xinrong Meng) > Memory Profiler on Executors >

[jira] [Created] (SPARK-40281) Memory Profiler on Executors

2022-08-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-40281: Summary: Memory Profiler on Executors Key: SPARK-40281 URL: https://issues.apache.org/jira/browse/SPARK-40281 Project: Spark Issue Type: Umbrella

[jira] [Assigned] (SPARK-40281) Memory Profiler on Executors

2022-08-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-40281: Assignee: Xinrong Meng > Memory Profiler on Executors > > >

[jira] [Closed] (SPARK-40266) Corrected console output in quick-start - Datatype Integer instead of Long

2022-08-30 Thread Prashant Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Singh closed SPARK-40266. -- > Corrected console output in quick-start - Datatype Integer instead of Long >

[jira] [Resolved] (SPARK-40256) Switch base image from openjdk to eclipse-temurin

2022-08-30 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-40256. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37705

[jira] [Assigned] (SPARK-40256) Switch base image from openjdk to eclipse-temurin

2022-08-30 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-40256: -- Assignee: Yikun Jiang > Switch base image from openjdk to eclipse-temurin >

[jira] [Commented] (SPARK-40264) Add helper function for DL model inference in pyspark.ml.functions

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598067#comment-17598067 ] Apache Spark commented on SPARK-40264: -- User 'leewyang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40264) Add helper function for DL model inference in pyspark.ml.functions

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40264: Assignee: Apache Spark > Add helper function for DL model inference in

[jira] [Assigned] (SPARK-40264) Add helper function for DL model inference in pyspark.ml.functions

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40264: Assignee: (was: Apache Spark) > Add helper function for DL model inference in

[jira] [Created] (SPARK-40280) Failure to create parquet predicate push down for ints and longs on some valid files

2022-08-30 Thread Robert Joseph Evans (Jira)
Robert Joseph Evans created SPARK-40280: --- Summary: Failure to create parquet predicate push down for ints and longs on some valid files Key: SPARK-40280 URL:

[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-30 Thread Niranda Perera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598021#comment-17598021 ] Niranda Perera commented on SPARK-40233: I believe the issue is related to executors not being

[jira] [Assigned] (SPARK-40267) Add description for ExecutorAllocationManager metrics

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40267: Assignee: Apache Spark > Add description for ExecutorAllocationManager metrics >

[jira] [Assigned] (SPARK-40267) Add description for ExecutorAllocationManager metrics

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40267: Assignee: (was: Apache Spark) > Add description for ExecutorAllocationManager

[jira] [Commented] (SPARK-40267) Add description for ExecutorAllocationManager metrics

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597996#comment-17597996 ] Apache Spark commented on SPARK-40267: -- User 'warrenzhu25' has created a pull request for this

[jira] [Resolved] (SPARK-40260) Use error classes in the compilation errors of GROUP BY a position

2022-08-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40260. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37712

[jira] [Commented] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597977#comment-17597977 ] Apache Spark commented on SPARK-40253: -- User 'SelfImpr001' has created a pull request for this

[jira] [Commented] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597975#comment-17597975 ] Apache Spark commented on SPARK-40253: -- User 'SelfImpr001' has created a pull request for this

[jira] [Resolved] (SPARK-38603) Qualified star selection produces duplicated common columns after join then alias

2022-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38603. - Resolution: Duplicate > Qualified star selection produces duplicated common columns after join

[jira] [Commented] (SPARK-31001) Add ability to create a partitioned table via catalog.createTable()

2022-08-30 Thread Kevin Appel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597941#comment-17597941 ] Kevin Appel commented on SPARK-31001: - [~nchammas]  I ran into this recently trying to create the

[jira] [Resolved] (SPARK-40113) Reactor ParquetScanBuilder DataSourceV2 interface implementation

2022-08-30 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-40113. Fix Version/s: 3.4.0 Assignee: miracle Resolution: Fixed > Reactor

[jira] [Resolved] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40056. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37727

[jira] [Assigned] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40056: Assignee: BingKun Pan > Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9 >

[jira] [Commented] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread yihangqiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597905#comment-17597905 ] yihangqiao commented on SPARK-40253: solution: In Literal, 1 significant digit reserved digit is

[jira] [Assigned] (SPARK-40279) Document spark.yarn.report.interval

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40279: Assignee: Apache Spark > Document spark.yarn.report.interval >

[jira] [Assigned] (SPARK-40279) Document spark.yarn.report.interval

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40279: Assignee: (was: Apache Spark) > Document spark.yarn.report.interval >

[jira] [Commented] (SPARK-40279) Document spark.yarn.report.interval

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597896#comment-17597896 ] Apache Spark commented on SPARK-40279: -- User 'LucaCanali' has created a pull request for this

[jira] [Created] (SPARK-40279) Document spark.yarn.report.interval

2022-08-30 Thread Luca Canali (Jira)
Luca Canali created SPARK-40279: --- Summary: Document spark.yarn.report.interval Key: SPARK-40279 URL: https://issues.apache.org/jira/browse/SPARK-40279 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597894#comment-17597894 ] Apache Spark commented on SPARK-39915: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597893#comment-17597893 ] Apache Spark commented on SPARK-39915: -- User 'ulysses-you' has created a pull request for this

[jira] [Updated] (SPARK-40278) Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed

2022-08-30 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40278: - Description: I used databricks spark-sql-pref + Spark 3.3 to run 3TB TPCDS q24a or q24b, the test code

[jira] [Created] (SPARK-40278) Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed

2022-08-30 Thread Yang Jie (Jira)
Yang Jie created SPARK-40278: Summary: Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed Key: SPARK-40278 URL: https://issues.apache.org/jira/browse/SPARK-40278 Project: Spark

[jira] [Updated] (SPARK-40278) Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed

2022-08-30 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40278: - Affects Version/s: (was: 3.4.0) > Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds

[jira] [Commented] (SPARK-33861) Simplify conditional in predicate

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597865#comment-17597865 ] Apache Spark commented on SPARK-33861: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-33861) Simplify conditional in predicate

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597862#comment-17597862 ] Apache Spark commented on SPARK-33861: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-40277) Use DataFrame's column for referring to DDL schema for from_csv() and from_json()

2022-08-30 Thread Jayant Kumar (Jira)
Jayant Kumar created SPARK-40277: Summary: Use DataFrame's column for referring to DDL schema for from_csv() and from_json() Key: SPARK-40277 URL: https://issues.apache.org/jira/browse/SPARK-40277

[jira] [Commented] (SPARK-40276) reduce the result size of RDD.takeOrdered

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597852#comment-17597852 ] Apache Spark commented on SPARK-40276: -- User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-40276) reduce the result size of RDD.takeOrdered

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597850#comment-17597850 ] Apache Spark commented on SPARK-40276: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-40276) reduce the result size of RDD.takeOrdered

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40276: Assignee: (was: Apache Spark) > reduce the result size of RDD.takeOrdered >

[jira] [Assigned] (SPARK-40276) reduce the result size of RDD.takeOrdered

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40276: Assignee: Apache Spark > reduce the result size of RDD.takeOrdered >

[jira] [Created] (SPARK-40276) reduce the result size of RDD.takeOrdered

2022-08-30 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40276: - Summary: reduce the result size of RDD.takeOrdered Key: SPARK-40276 URL: https://issues.apache.org/jira/browse/SPARK-40276 Project: Spark Issue Type:

[jira] [Commented] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597778#comment-17597778 ] Apache Spark commented on SPARK-40056: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-40207) Specify the column name when the data type is not supported by datasource

2022-08-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-40207: --- Assignee: Yi kaifei (was: Apache Spark) > Specify the column name when the data type is

[jira] [Resolved] (SPARK-40207) Specify the column name when the data type is not supported by datasource

2022-08-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40207. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37574

[jira] [Created] (SPARK-40275) Support casting decimal128

2022-08-30 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-40275: -- Summary: Support casting decimal128 Key: SPARK-40275 URL: https://issues.apache.org/jira/browse/SPARK-40275 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Environment: spark 3.1.2 scala 2.12.10 jdk 11 linux (was: spark 3.1.2 scala 2.12.10 jdk 1.8 linux) >

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Description: spark 3.1.2 scala 2.12.10 jdk 1.8 linux   when use dataframe.count will throw this exception:  

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Environment: spark 3.1.2 scala 2.12.10 jdk 1.8 linux (was:             com.fasterxml.jackson.core            

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Attachment: pom.txt > ArrayIndexOutOfBoundsException in BytecodeReadingParanamer >

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Attachment: code.scala > ArrayIndexOutOfBoundsException in BytecodeReadingParanamer >

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Attachment: error.txt > ArrayIndexOutOfBoundsException in BytecodeReadingParanamer >

[jira] [Updated] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张刘强 updated SPARK-40274: Docs Text: (was: code like this: val dataFrame: DataFrame = sparkSession.read .format(JDBC)

[jira] [Created] (SPARK-40274) ArrayIndexOutOfBoundsException in BytecodeReadingParanamer

2022-08-30 Thread Jira
张刘强 created SPARK-40274: --- Summary: ArrayIndexOutOfBoundsException in BytecodeReadingParanamer Key: SPARK-40274 URL: https://issues.apache.org/jira/browse/SPARK-40274 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40253: Assignee: Apache Spark > Data read exception in orc format >

[jira] [Assigned] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40253: Assignee: (was: Apache Spark) > Data read exception in orc format >

[jira] [Commented] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597718#comment-17597718 ] Apache Spark commented on SPARK-40253: -- User 'SelfImpr001' has created a pull request for this

[jira] [Commented] (SPARK-40253) Data read exception in orc format

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597717#comment-17597717 ] Apache Spark commented on SPARK-40253: -- User 'SelfImpr001' has created a pull request for this

[jira] [Assigned] (SPARK-40273) Fix the documents "Contributing and Maintaining Type Hints".

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40273: Assignee: Apache Spark > Fix the documents "Contributing and Maintaining Type Hints". >

[jira] [Commented] (SPARK-40273) Fix the documents "Contributing and Maintaining Type Hints".

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597690#comment-17597690 ] Apache Spark commented on SPARK-40273: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40273) Fix the documents "Contributing and Maintaining Type Hints".

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40273: Assignee: (was: Apache Spark) > Fix the documents "Contributing and Maintaining Type

[jira] [Commented] (SPARK-39763) Executor memory footprint substantially increases while reading zstd compressed parquet files

2022-08-30 Thread Fengyu Cao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597685#comment-17597685 ] Fengyu Cao commented on SPARK-39763: had the same problem   one of our dataset, 75GB in zstd

[jira] [Commented] (SPARK-40273) Fix the documents "Contributing and Maintaining Type Hints".

2022-08-30 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597646#comment-17597646 ] Haejoon Lee commented on SPARK-40273: - I'm working on this > Fix the documents "Contributing and

[jira] [Created] (SPARK-40273) Fix the documents "Contributing and Maintaining Type Hints".

2022-08-30 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-40273: --- Summary: Fix the documents "Contributing and Maintaining Type Hints". Key: SPARK-40273 URL: https://issues.apache.org/jira/browse/SPARK-40273 Project: Spark

[jira] [Commented] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597637#comment-17597637 ] Apache Spark commented on SPARK-40271: -- User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597634#comment-17597634 ] Apache Spark commented on SPARK-40271: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40271: Assignee: Apache Spark > Support list type for pyspark.sql.functions.lit >

[jira] [Assigned] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40271: Assignee: (was: Apache Spark) > Support list type for pyspark.sql.functions.lit >

[jira] [Assigned] (SPARK-40266) Corrected console output in quick-start - Datatype Integer instead of Long

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40266: Assignee: Prashant Singh > Corrected console output in quick-start - Datatype Integer

[jira] [Resolved] (SPARK-40266) Corrected console output in quick-start - Datatype Integer instead of Long

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40266. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37719

[jira] [Updated] (SPARK-40271) Support list type for pyspark.sql.functions.lit

2022-08-30 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40271: Summary: Support list type for pyspark.sql.functions.lit (was: Support list type for

[jira] [Assigned] (SPARK-40270) Make compute.max_rows as None working in DataFrame.style

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40270: Assignee: Hyukjin Kwon > Make compute.max_rows as None working in DataFrame.style >

[jira] [Resolved] (SPARK-40270) Make compute.max_rows as None working in DataFrame.style

2022-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40270. -- Fix Version/s: (was: 3.1.4) Resolution: Fixed Issue resolved by pull request 37718

  1   2   >