[jira] [Created] (SPARK-42991) Disable string +/- interval in ANSI mode

2023-03-30 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42991: --- Summary: Disable string +/- interval in ANSI mode Key: SPARK-42991 URL: https://issues.apache.org/jira/browse/SPARK-42991 Project: Spark Issue Type: Improvemen

[jira] [Commented] (SPARK-42972) ExecutorAllocationManager cannot allocate new instances when all executors down.

2023-03-30 Thread liang yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707143#comment-17707143 ] liang yu commented on SPARK-42972: -- When the ratio is lower than the scalingDownRatio,

[jira] [Created] (SPARK-42990) Apply error framework to UnsupportedOperationChecker

2023-03-30 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-42990: Summary: Apply error framework to UnsupportedOperationChecker Key: SPARK-42990 URL: https://issues.apache.org/jira/browse/SPARK-42990 Project: Spark Issue Ty

[jira] [Commented] (SPARK-42760) The partition of result data frame of join is always 1

2023-03-30 Thread Ranga Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707125#comment-17707125 ] Ranga Reddy commented on SPARK-42760: - >From Spark version 3.2.0 onwards AQE is enab

[jira] [Created] (SPARK-42989) When Spark is going to end support of Hadoop2

2023-03-30 Thread shamim (Jira)
shamim created SPARK-42989: -- Summary: When Spark is going to end support of Hadoop2 Key: SPARK-42989 URL: https://issues.apache.org/jira/browse/SPARK-42989 Project: Spark Issue Type: Question

[jira] [Commented] (SPARK-42988) Spark Sql insert into hive table dynamic partitions slow

2023-03-30 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707116#comment-17707116 ] thomasgx commented on SPARK-42988: -- saprk using  hive api externalCatalog.loadDynamicP

[jira] [Commented] (SPARK-42932) Spark 3.3.2, with hadoop3, Error with java.io.IOException: Mkdirs failed to create file

2023-03-30 Thread shamim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707106#comment-17707106 ] shamim commented on SPARK-42932: its working fine with hadoop2, getting issue when using

[jira] [Assigned] (SPARK-41391) The output column name of `groupBy.agg(count_distinct)` is incorrect

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-41391: --- Assignee: Ritika Maheshwari > The output column name of `groupBy.agg(count_distinct)` is in

[jira] [Resolved] (SPARK-41391) The output column name of `groupBy.agg(count_distinct)` is incorrect

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41391. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40116 [https://gith

[jira] [Commented] (SPARK-42988) Spark Sql insert into hive table dynamic partitions slow

2023-03-30 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707096#comment-17707096 ] thomasgx commented on SPARK-42988: -- Use the same sql to execute on hive and spark, inse

[jira] [Created] (SPARK-42988) Spark Sql insert into hive table dynamic partitions slow

2023-03-30 Thread thomasgx (Jira)
thomasgx created SPARK-42988: Summary: Spark Sql insert into hive table dynamic partitions slow Key: SPARK-42988 URL: https://issues.apache.org/jira/browse/SPARK-42988 Project: Spark Issue Type:

[jira] [Commented] (SPARK-42987) Correct code highlights in SQL protobuf documentation

2023-03-30 Thread Lucas Pompeu Neves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707082#comment-17707082 ] Lucas Pompeu Neves commented on SPARK-42987: The PR was created in github h

[jira] [Updated] (SPARK-42987) Correct code highlights in SQL protobuf documentation

2023-03-30 Thread Lucas Pompeu Neves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Pompeu Neves updated SPARK-42987: --- Summary: Correct code highlights in SQL protobuf documentation (was: [SPARK-42987][

[jira] [Updated] (SPARK-42987) [SPARK-42987][DOCS] Correct code highlights in SQL protobuf documentation

2023-03-30 Thread Lucas Pompeu Neves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Pompeu Neves updated SPARK-42987: --- Summary: [SPARK-42987][DOCS] Correct code highlights in SQL protobuf documentation

[jira] [Created] (SPARK-42987) [SPARK-xxxx][DOCS] Correct code highlights in SQL protobuf documentation

2023-03-30 Thread Lucas Pompeu Neves (Jira)
Lucas Pompeu Neves created SPARK-42987: -- Summary: [SPARK-][DOCS] Correct code highlights in SQL protobuf documentation Key: SPARK-42987 URL: https://issues.apache.org/jira/browse/SPARK-42987

[jira] [Created] (SPARK-42986) Migrate more PySpark errors onto error classes

2023-03-30 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-42986: --- Summary: Migrate more PySpark errors onto error classes Key: SPARK-42986 URL: https://issues.apache.org/jira/browse/SPARK-42986 Project: Spark Issue Type: Umbr

[jira] [Resolved] (SPARK-42969) Fix the comparison the result with Arrow optimization enabled/disabled.

2023-03-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42969. -- Assignee: Takuya Ueshin Resolution: Fixed Fixed in https://github.com/apache/spark/pull/

[jira] [Updated] (SPARK-42970) Reuse pyspark.sql.tests.test_arrow test cases

2023-03-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-42970: - Fix Version/s: 3.4.1 (was: 3.5.0) > Reuse pyspark.sql.tests.test_arrow te

[jira] [Resolved] (SPARK-42967) Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled

2023-03-30 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-42967. Fix Version/s: 3.3.3 3.2.4 3.4.0 Resolution: F

[jira] [Created] (SPARK-42985) Fix createDataFrame from pandas to respect session timezone.

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42985: - Summary: Fix createDataFrame from pandas to respect session timezone. Key: SPARK-42985 URL: https://issues.apache.org/jira/browse/SPARK-42985 Project: Spark

[jira] [Created] (SPARK-42984) Fix test_createDataFrame_with_single_data_type.

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42984: - Summary: Fix test_createDataFrame_with_single_data_type. Key: SPARK-42984 URL: https://issues.apache.org/jira/browse/SPARK-42984 Project: Spark Issue Type:

[jira] [Created] (SPARK-42983) Fix the error message of createDataFrame from np.array(0)

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42983: - Summary: Fix the error message of createDataFrame from np.array(0) Key: SPARK-42983 URL: https://issues.apache.org/jira/browse/SPARK-42983 Project: Spark I

[jira] [Created] (SPARK-42982) Fix createDataFrame from pandas with map type

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42982: - Summary: Fix createDataFrame from pandas with map type Key: SPARK-42982 URL: https://issues.apache.org/jira/browse/SPARK-42982 Project: Spark Issue Type: S

[jira] [Created] (SPARK-42981) Add direct arrow serialization

2023-03-30 Thread Jira
Herman van Hövell created SPARK-42981: - Summary: Add direct arrow serialization Key: SPARK-42981 URL: https://issues.apache.org/jira/browse/SPARK-42981 Project: Spark Issue Type: New Feat

[jira] [Created] (SPARK-42980) SmallBroadcast

2023-03-30 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-42980: Summary: SmallBroadcast Key: SPARK-42980 URL: https://issues.apache.org/jira/browse/SPARK-42980 Project: Spark Issue Type: New Feature Components: Spark Co

[jira] [Updated] (SPARK-42980) lightweight Broadcast implementation

2023-03-30 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-42980: - Summary: lightweight Broadcast implementation (was: SmallBroadcast) > lightweight Broadcast implementat

[jira] [Updated] (SPARK-42976) spark sql Disable vectorized faild

2023-03-30 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-42976: Issue Type: Improvement (was: Question) > spark sql Disable vectorized faild > -

[jira] [Commented] (SPARK-35198) Add support for calling debugCodegen from Python & Java

2023-03-30 Thread Juanvi Soler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706926#comment-17706926 ] Juanvi Soler commented on SPARK-35198: -- Ended up creating a PR https://github.com/a

[jira] [Resolved] (SPARK-42979) Define literal constructors as keywords

2023-03-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-42979. -- Resolution: Fixed Issue resolved by pull request 40593 [https://github.com/apache/spark/pull/40593] >

[jira] [Updated] (SPARK-42979) Define literal constructors as keywords

2023-03-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-42979: - Description: Currently, Spark SQL defines literal constructors as:   {code} | identifier stringLit #typ

[jira] [Created] (SPARK-42979) Define literal constructors as keywords

2023-03-30 Thread Max Gekk (Jira)
Max Gekk created SPARK-42979: Summary: Define literal constructors as keywords Key: SPARK-42979 URL: https://issues.apache.org/jira/browse/SPARK-42979 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-42316) Assign name to _LEGACY_ERROR_TEMP_2044

2023-03-30 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706865#comment-17706865 ] Jia Fan commented on SPARK-42316: - I'm working on it > Assign name to _LEGACY_ERROR_TEM

[jira] [Assigned] (SPARK-42968) Add option to skip using commit coordinator in StreamingWrite API

2023-03-30 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-42968: Assignee: Anish Shrigondekar > Add option to skip using commit coordinator in StreamingWr

[jira] [Resolved] (SPARK-42968) Add option to skip using commit coordinator in StreamingWrite API

2023-03-30 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-42968. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40600 [https://gi

[jira] [Comment Edited] (SPARK-35198) Add support for calling debugCodegen from Python & Java

2023-03-30 Thread Juanvi Soler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706493#comment-17706493 ] Juanvi Soler edited comment on SPARK-35198 at 3/30/23 12:30 PM: --

[jira] [Reopened] (SPARK-41765) Pull out v1 write metrics to WriteFiles

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-41765: - Assignee: (was: XiDuo You) > Pull out v1 write metrics to WriteFiles > ---

[jira] [Resolved] (SPARK-41765) Pull out v1 write metrics to WriteFiles

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41765. - Fix Version/s: (was: 3.4.0) Resolution: Won't Do > Pull out v1 write metrics to Write

[jira] [Commented] (SPARK-42978) Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706793#comment-17706793 ] ASF GitHub Bot commented on SPARK-42978: User 'yaooqinn' has created a pull requ

[jira] [Assigned] (SPARK-42683) Automatically rename metadata columns that conflict with data schema columns

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42683: --- Assignee: Ryan Johnson > Automatically rename metadata columns that conflict with data sche

[jira] [Commented] (SPARK-42958) Refactor `CheckConnectJvmClientCompatibility` to compare client and avro

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706787#comment-17706787 ] ASF GitHub Bot commented on SPARK-42958: User 'LuciferYang' has created a pull r

[jira] [Resolved] (SPARK-42683) Automatically rename metadata columns that conflict with data schema columns

2023-03-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42683. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40300 [https://gith

[jira] [Updated] (SPARK-42974) Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup tempDir

2023-03-30 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-42974: - Issue Type: Improvement (was: Bug) > Restore `Utils#createTempDir` use > `ShutdownHookManager.registe