[jira] [Created] (SPARK-40844) Flip the default value of Kafka offset fetching config

2022-10-18 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-40844: Summary: Flip the default value of Kafka offset fetching config Key: SPARK-40844 URL: https://issues.apache.org/jira/browse/SPARK-40844 Project: Spark Issue

[jira] [Assigned] (SPARK-40843) Clean up deprecated api usage in SparkThrowableSuite

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40843: Assignee: (was: Apache Spark) > Clean up deprecated api usage in SparkThrowableSuite

[jira] [Commented] (SPARK-40843) Clean up deprecated api usage in SparkThrowableSuite

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620023#comment-17620023 ] Apache Spark commented on SPARK-40843: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-40843) Clean up deprecated api usage in SparkThrowableSuite

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40843: Assignee: Apache Spark > Clean up deprecated api usage in SparkThrowableSuite > -

[jira] [Updated] (SPARK-35490) Update json4s to 3.7.0-M11

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-35490: - Priority: Trivial (was: Minor) > Update json4s to 3.7.0-M11 > -- > >

[jira] [Assigned] (SPARK-40811) Use checkError() to intercept ParseException

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40811: Assignee: Max Gekk > Use checkError() to intercept ParseException > -

[jira] [Resolved] (SPARK-40811) Use checkError() to intercept ParseException

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40811. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38267 [https://github.com

[jira] [Created] (SPARK-40843) Clean up deprecated api usage in SparkThrowableSuite

2022-10-18 Thread Yang Jie (Jira)
Yang Jie created SPARK-40843: Summary: Clean up deprecated api usage in SparkThrowableSuite Key: SPARK-40843 URL: https://issues.apache.org/jira/browse/SPARK-40843 Project: Spark Issue Type: Impr

[jira] [Assigned] (SPARK-40810) Use SparkIllegalArgumentException instead of IllegalArgumentException in CreateDatabaseCommand & AlterDatabaseSetLocationCommand

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40810: Assignee: BingKun Pan > Use SparkIllegalArgumentException instead of IllegalArgumentException in

[jira] [Resolved] (SPARK-40810) Use SparkIllegalArgumentException instead of IllegalArgumentException in CreateDatabaseCommand & AlterDatabaseSetLocationCommand

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40810. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38274 [https://github.com

[jira] [Updated] (SPARK-40839) [Python] Implement `DataFrame.sample`

2022-10-18 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-40839: - Summary: [Python] Implement `DataFrame.sample` (was: Implement `DataFrame.sample`) > [Python] Implemen

[jira] [Created] (SPARK-40842) Support s3 and hdfs paths for keytab file while being passed

2022-10-18 Thread Anirudh Vyas (Jira)
Anirudh Vyas created SPARK-40842: Summary: Support s3 and hdfs paths for keytab file while being passed Key: SPARK-40842 URL: https://issues.apache.org/jira/browse/SPARK-40842 Project: Spark

[jira] [Created] (SPARK-40841) Support k8s secret as keytab file when using kerberos on spark on kubernetes

2022-10-18 Thread Anirudh Vyas (Jira)
Anirudh Vyas created SPARK-40841: Summary: Support k8s secret as keytab file when using kerberos on spark on kubernetes Key: SPARK-40841 URL: https://issues.apache.org/jira/browse/SPARK-40841 Project:

[jira] [Created] (SPARK-40840) Support Keytab file passing as a k8s secret

2022-10-18 Thread Anirudh Vyas (Jira)
Anirudh Vyas created SPARK-40840: Summary: Support Keytab file passing as a k8s secret Key: SPARK-40840 URL: https://issues.apache.org/jira/browse/SPARK-40840 Project: Spark Issue Type: New F

[jira] [Commented] (SPARK-38390) Spark submit k8s with proxy user

2022-10-18 Thread Anirudh Vyas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620005#comment-17620005 ] Anirudh Vyas commented on SPARK-38390: -- This code should work on driver  {code:java

[jira] [Resolved] (SPARK-40826) Add additional checkpoint rename file check

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40826. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38291 [https://

[jira] [Assigned] (SPARK-40826) Add additional checkpoint rename file check

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40826: - Assignee: L. C. Hsieh > Add additional checkpoint rename file check > -

[jira] (SPARK-38390) Spark submit k8s with proxy user

2022-10-18 Thread Anirudh Vyas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38390 ] Anirudh Vyas deleted comment on SPARK-38390: -- was (Author: JIRAUSER297214): use  {code:java} --conf spark.kerberos.keytab=local://keytab file {code} Mount the keytab file on `/tmp` and it sh

[jira] [Commented] (SPARK-38390) Spark submit k8s with proxy user

2022-10-18 Thread Anirudh Vyas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620003#comment-17620003 ] Anirudh Vyas commented on SPARK-38390: -- use  {code:java} --conf spark.kerberos.keyt

[jira] [Assigned] (SPARK-40839) Implement `DataFrame.sample`

2022-10-18 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40839: - Assignee: Ruifeng Zheng > Implement `DataFrame.sample` > >

[jira] [Created] (SPARK-40839) Implement `DataFrame.sample`

2022-10-18 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40839: - Summary: Implement `DataFrame.sample` Key: SPARK-40839 URL: https://issues.apache.org/jira/browse/SPARK-40839 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-40303) The performance will be worse after codegen

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619978#comment-17619978 ] Yuming Wang commented on SPARK-40303: - How to run benchmark code: # Download lates

[jira] [Commented] (SPARK-39658) Reconsider exposure of Json4s symbols in public ResourceInformation.toJson API

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619964#comment-17619964 ] Yang Jie commented on SPARK-39658: -- Do we have any clear conclusions or plans for this

[jira] [Resolved] (SPARK-40655) Protobuf functions in Python

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40655. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38212 [https://gi

[jira] [Commented] (SPARK-40838) Upgrade infra base image to focal-20220922

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619955#comment-17619955 ] Apache Spark commented on SPARK-40838: -- User 'Yikun' has created a pull request for

[jira] [Assigned] (SPARK-40838) Upgrade infra base image to focal-20220922

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40838: Assignee: Apache Spark > Upgrade infra base image to focal-20220922 > ---

[jira] [Assigned] (SPARK-40838) Upgrade infra base image to focal-20220922

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40838: Assignee: (was: Apache Spark) > Upgrade infra base image to focal-20220922 >

[jira] [Commented] (SPARK-40838) Upgrade infra base image to focal-20220922

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619956#comment-17619956 ] Apache Spark commented on SPARK-40838: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-40538) Add missing PySpark functions to Spark Connect

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619954#comment-17619954 ] Apache Spark commented on SPARK-40538: -- User 'HyukjinKwon' has created a pull reque

[jira] [Resolved] (SPARK-40823) Connect Proto should carry unparsed identifiers

2022-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40823. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38264 [https://gith

[jira] [Assigned] (SPARK-40823) Connect Proto should carry unparsed identifiers

2022-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40823: --- Assignee: Rui Wang > Connect Proto should carry unparsed identifiers >

[jira] [Created] (SPARK-40838) Upgrade infra base image to focal-20220922

2022-10-18 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-40838: --- Summary: Upgrade infra base image to focal-20220922 Key: SPARK-40838 URL: https://issues.apache.org/jira/browse/SPARK-40838 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-40833) Cleanup apt lists cache in Dockerfile

2022-10-18 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-40833. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 14 [https://github.

[jira] [Assigned] (SPARK-40833) Cleanup apt lists cache in Dockerfile

2022-10-18 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang reassigned SPARK-40833: --- Assignee: Yikun Jiang > Cleanup apt lists cache in Dockerfile > ---

[jira] [Resolved] (SPARK-40832) Add README for spark-docker

2022-10-18 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-40832. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 13 [https://github.

[jira] [Assigned] (SPARK-40832) Add README for spark-docker

2022-10-18 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang reassigned SPARK-40832: --- Assignee: Yikun Jiang > Add README for spark-docker > --- > >

[jira] [Resolved] (SPARK-40831) LogicalPlanStats is throwing an exception

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40831. -- Resolution: Invalid > LogicalPlanStats is throwing an exception > ---

[jira] [Commented] (SPARK-40831) LogicalPlanStats is throwing an exception

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619933#comment-17619933 ] Hyukjin Kwon commented on SPARK-40831: -- [~rohan997] these logical plans are not API

[jira] [Resolved] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40835. -- Resolution: Duplicate > to_utc_timestamp creates null column > ---

[jira] [Resolved] (SPARK-40367) Total size of serialized results of 3730 tasks (64.0 GB) is bigger than spark.driver.maxResultSize (64.0 GB)

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40367. -- Resolution: Not A Problem > Total size of serialized results of 3730 tasks (64.0 GB) is bigge

[jira] [Updated] (SPARK-40367) Total size of serialized results of 3730 tasks (64.0 GB) is bigger than spark.driver.maxResultSize (64.0 GB)

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40367: - Priority: Major (was: Blocker) > Total size of serialized results of 3730 tasks (64.0 GB) is b

[jira] [Updated] (SPARK-40804) Missing handling a catalog name in destination tables in `RenameTableExec`

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40804: - Priority: Major (was: Blocker) > Missing handling a catalog name in destination tables in `Rena

[jira] [Commented] (SPARK-40755) Migrate type check failures of number formatting onto error classes

2022-10-18 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619931#comment-17619931 ] BingKun Pan commented on SPARK-40755: - I will work on it. > Migrate type check fail

[jira] [Assigned] (SPARK-40834) Use SparkListenerSQLExecutionEnd to track final SQL status in UI

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40834: Assignee: (was: Apache Spark) > Use SparkListenerSQLExecutionEnd to track final SQL s

[jira] [Assigned] (SPARK-40834) Use SparkListenerSQLExecutionEnd to track final SQL status in UI

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40834: Assignee: Apache Spark > Use SparkListenerSQLExecutionEnd to track final SQL status in UI

[jira] [Commented] (SPARK-40834) Use SparkListenerSQLExecutionEnd to track final SQL status in UI

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619928#comment-17619928 ] Apache Spark commented on SPARK-40834: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2022-10-18 Thread jiangjian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619927#comment-17619927 ] jiangjian commented on SPARK-28921: --- [~thesuperzapper] Where can I get these two jar p

[jira] [Updated] (SPARK-40814) Exception in thread "main" java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/KubernetesClient

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40814: - Affects Version/s: 3.1.2 > Exception in thread "main" java.lang.NoClassDefFoundError: > io/fabr

[jira] [Assigned] (SPARK-40816) Python: rename LogicalPlan.collect to LogicalPlan.to_proto

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40816: Assignee: Rui Wang > Python: rename LogicalPlan.collect to LogicalPlan.to_proto > ---

[jira] [Resolved] (SPARK-40816) Python: rename LogicalPlan.collect to LogicalPlan.to_proto

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40816. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38279 [https://gi

[jira] [Resolved] (SPARK-40828) Drop Python test tables before and after unit tests

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40828. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38293 [https://gi

[jira] [Assigned] (SPARK-40828) Drop Python test tables before and after unit tests

2022-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40828: Assignee: Rui Wang > Drop Python test tables before and after unit tests > --

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619910#comment-17619910 ] Yang Jie commented on SPARK-40791: -- Okay, let's keep it as it is   > The semantics of

[jira] [Commented] (SPARK-40837) Structured Streaming on Dataproc - suddenly giving error while writing to Kafka topic

2022-10-18 Thread Karan Alang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619901#comment-17619901 ] Karan Alang commented on SPARK-40837: - Pls note - I'm able to read/write from topic

[jira] [Updated] (SPARK-40829) STORED AS serde in CREATE TABLE LIKE view does not work

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40829: -- Affects Version/s: 3.2.2 3.3.0 > STORED AS serde in CREATE TABLE LIKE v

[jira] [Updated] (SPARK-40829) STORED AS serde in CREATE TABLE LIKE view does not work

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40829: -- Issue Type: Bug (was: Improvement) > STORED AS serde in CREATE TABLE LIKE view does not work

[jira] [Updated] (SPARK-40837) Structured Streaming on Dataproc - suddenly giving error while writing to Kafka topic

2022-10-18 Thread Karan Alang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Alang updated SPARK-40837: Environment: where i'm running Structured Streaming job - Dataproc Source Kafka topic - VM on GCP

[jira] [Created] (SPARK-40837) Structured Streaming on Dataproc - suddenly giving error while writing to Kafka topic

2022-10-18 Thread Karan Alang (Jira)
Karan Alang created SPARK-40837: --- Summary: Structured Streaming on Dataproc - suddenly giving error while writing to Kafka topic Key: SPARK-40837 URL: https://issues.apache.org/jira/browse/SPARK-40837 P

[jira] [Resolved] (SPARK-40829) STORED AS serde in CREATE TABLE LIKE view does not work

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40829. --- Fix Version/s: 3.3.2 3.2.3 3.4.0 Resolution: Fix

[jira] [Assigned] (SPARK-40829) STORED AS serde in CREATE TABLE LIKE view does not work

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40829: - Assignee: zhangbutao > STORED AS serde in CREATE TABLE LIKE view does not work > --

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619796#comment-17619796 ] Dongjoon Hyun commented on SPARK-40791: --- Thank you for pinging me, but let's not f

[jira] [Assigned] (SPARK-40836) AnalyzeResult should use struct for schema

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40836: Assignee: Apache Spark > AnalyzeResult should use struct for schema > ---

[jira] [Assigned] (SPARK-40836) AnalyzeResult should use struct for schema

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40836: Assignee: (was: Apache Spark) > AnalyzeResult should use struct for schema >

[jira] [Commented] (SPARK-40836) AnalyzeResult should use struct for schema

2022-10-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619783#comment-17619783 ] Apache Spark commented on SPARK-40836: -- User 'amaliujia' has created a pull request

[jira] [Created] (SPARK-40836) AnalyzeResult should use struct for schema

2022-10-18 Thread Rui Wang (Jira)
Rui Wang created SPARK-40836: Summary: AnalyzeResult should use struct for schema Key: SPARK-40836 URL: https://issues.apache.org/jira/browse/SPARK-40836 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Comment Edited] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619761#comment-17619761 ] Rohan Barman edited comment on SPARK-40835 at 10/18/22 7:56 PM: --

[jira] [Commented] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619761#comment-17619761 ] Rohan Barman commented on SPARK-40835: -- I found a similar issue https://issues.apac

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Commented] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619715#comment-17619715 ] xsys commented on SPARK-40637: -- Thank you for the response [~ivan.sadikov]. I just updated

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Summary: Spark-shell can correctly encode BINARY type but Spark-sql cannot (was: DataFrame can correctly encode

[jira] [Updated] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Barman updated SPARK-40835: - Description: We are in the process of migrating our PySpark applications from Spark version 3.1

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug When we store a BINARY value (e.g. {{BigInt("1").toByteArray)}} /  {{{}X'01'{

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug Storing a BINARY value (e.g. {{BigInt("1").toByteArray)}} / {{{}X'01'{}}}) vi

[jira] [Created] (SPARK-40835) to_utc_timestamp creates null column

2022-10-18 Thread Rohan Barman (Jira)
Rohan Barman created SPARK-40835: Summary: to_utc_timestamp creates null column Key: SPARK-40835 URL: https://issues.apache.org/jira/browse/SPARK-40835 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug Storing a BINARY value (e.g. {{BigInt("1").toByteArray)}} / {{{}X'01'{}}}) vi

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug Storing a BINARY value (e.g. {{BigInt("1").toByteArray)}} / {{{}X'01'{}}}) vi

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug Storing a BINARY value (e.g. {{BigInt("1").toByteArray)}} / {{{}X'01'{}}}) vi

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Attachment: image-2022-10-18-12-15-05-576.png > DataFrame can correctly encode BINARY type but SparkSQL cannot >

[jira] [Updated] (SPARK-40637) DataFrame can correctly encode BINARY type but SparkSQL cannot

2022-10-18 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-40637: - Description: h3. Describe the bug Storing a BINARY value (e.g. {{BigInt("1").toByteArray)}} / {{{}X'01'{}}}) vi

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619681#comment-17619681 ] Yang Jie commented on SPARK-40791: -- Yes, when run test in `datetime-formatting.sql` wit

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619663#comment-17619663 ] Sean R. Owen commented on SPARK-40791: -- Well it seems like Java has changed its min

[jira] [Comment Edited] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619653#comment-17619653 ] Yang Jie edited comment on SPARK-40791 at 10/18/22 3:30 PM:

[jira] [Comment Edited] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619653#comment-17619653 ] Yang Jie edited comment on SPARK-40791 at 10/18/22 3:26 PM:

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619653#comment-17619653 ] Yang Jie commented on SPARK-40791: -- Also cc [~srowen] [~rednaxelafx] [~dongjoon]  [~clo

[jira] [Comment Edited] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619613#comment-17619613 ] Yang Jie edited comment on SPARK-40791 at 10/18/22 3:15 PM:

[jira] [Resolved] (SPARK-40369) Migrate the type check failures of calls via reflection onto error classes

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40369. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38294 [https://github.com

[jira] [Assigned] (SPARK-40369) Migrate the type check failures of calls via reflection onto error classes

2022-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40369: Assignee: Yang Jie > Migrate the type check failures of calls via reflection onto error classes >

[jira] [Updated] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40791: - Summary: The semantics of `F` in `DateTimeFormatter` have changed (was: Result of `select col, date_for

[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619613#comment-17619613 ] Yang Jie commented on SPARK-40791: -- [~hyukjin.kwon]  I found that the difference was c

[jira] [Updated] (SPARK-40791) Result of `select col, date_format(col, 'F') from v;` in datetime-formatting.sql different from other versions

2022-10-18 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40791: - Description: {code:java} val createSql = """ |create temporary view v as select col from values

[jira] [Resolved] (SPARK-40615) Check unsupported data type when decorrelating subqueries

2022-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40615. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38050 [https://gith

  1   2   >