[jira] [Resolved] (SPARK-34701) Remove analyzing temp view again in CreateViewCommand

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34701. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31933 [https://gith

[jira] [Assigned] (SPARK-34701) Remove analyzing temp view again in CreateViewCommand

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34701: --- Assignee: Terry Kim > Remove analyzing temp view again in CreateViewCommand > -

[jira] [Commented] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308435#comment-17308435 ] Yuming Wang commented on SPARK-34859: - [~lxian2] Thank you for reporting this issue.

[jira] [Resolved] (SPARK-34850) Multiply day-time interval by numeric

2021-03-25 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-34850. -- Resolution: Fixed Issue resolved by pull request 31951 [https://github.com/apache/spark/pull/31951] >

[jira] [Comment Edited] (SPARK-34345) Allow several properties files

2021-03-25 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307644#comment-17307644 ] Arseniy Tashoyan edited comment on SPARK-34345 at 3/25/21, 7:52 AM: --

[jira] [Resolved] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34857. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31956 [https://gi

[jira] [Assigned] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34857: Assignee: Tim Armstrong > AtLeastNNonNulls does not show up correctly in explain > --

[jira] [Updated] (SPARK-34861) Support nested column in Spark vectorized readers

2021-03-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-34861: - Summary: Support nested column in Spark vectorized readers (was: Support nested column in spark vectori

[jira] [Created] (SPARK-34861) Support nested column in spark vectorized reader

2021-03-25 Thread Cheng Su (Jira)
Cheng Su created SPARK-34861: Summary: Support nested column in spark vectorized reader Key: SPARK-34861 URL: https://issues.apache.org/jira/browse/SPARK-34861 Project: Spark Issue Type: Umbrella

[jira] [Commented] (SPARK-34861) Support nested column in Spark vectorized readers

2021-03-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308501#comment-17308501 ] Cheng Su commented on SPARK-34861: -- Just FYI we are working on each individual tasks no

[jira] [Created] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Cheng Su (Jira)
Cheng Su created SPARK-34862: Summary: Support nested column in Spark ORC vectorized readers Key: SPARK-34862 URL: https://issues.apache.org/jira/browse/SPARK-34862 Project: Spark Issue Type: Sub

[jira] [Updated] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-34862: - Description: The task is to support nested column type in Spark ORC vectorized reader. Currently ORC ve

[jira] [Commented] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308504#comment-17308504 ] Cheng Su commented on SPARK-34862: -- Just FYI I am working on it now. > Support nested

[jira] [Created] (SPARK-34863) Support nested column in Spark Parquet vectorized readers

2021-03-25 Thread Cheng Su (Jira)
Cheng Su created SPARK-34863: Summary: Support nested column in Spark Parquet vectorized readers Key: SPARK-34863 URL: https://issues.apache.org/jira/browse/SPARK-34863 Project: Spark Issue Type:

[jira] [Commented] (SPARK-34863) Support nested column in Spark Parquet vectorized readers

2021-03-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308505#comment-17308505 ] Cheng Su commented on SPARK-34863: -- Just FYI I am working on it now. > Support nested

[jira] [Assigned] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34862: Assignee: (was: Apache Spark) > Support nested column in Spark ORC vectorized readers

[jira] [Assigned] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34862: Assignee: Apache Spark > Support nested column in Spark ORC vectorized readers >

[jira] [Resolved] (SPARK-30497) migrate DESCRIBE TABLE to the new framework

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30497. - Fix Version/s: 3.1.1 Resolution: Fixed > migrate DESCRIBE TABLE to the new framework > --

[jira] [Commented] (SPARK-34862) Support nested column in Spark ORC vectorized readers

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308524#comment-17308524 ] Apache Spark commented on SPARK-34862: -- User 'c21' has created a pull request for t

[jira] [Created] (SPARK-34864) cannot resolve methods when & col in spark java

2021-03-25 Thread unical1988 (Jira)
unical1988 created SPARK-34864: -- Summary: cannot resolve methods when & col in spark java Key: SPARK-34864 URL: https://issues.apache.org/jira/browse/SPARK-34864 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-34865) cannot resolve methods when & col in spark java

2021-03-25 Thread unical1988 (Jira)
unical1988 created SPARK-34865: -- Summary: cannot resolve methods when & col in spark java Key: SPARK-34865 URL: https://issues.apache.org/jira/browse/SPARK-34865 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-34866) cannot resolve method when of Column() spark java

2021-03-25 Thread unical1988 (Jira)
unical1988 created SPARK-34866: -- Summary: cannot resolve method when of Column() spark java Key: SPARK-34866 URL: https://issues.apache.org/jira/browse/SPARK-34866 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-34867) cannot resolve method when of Column() spark java

2021-03-25 Thread unical1988 (Jira)
unical1988 created SPARK-34867: -- Summary: cannot resolve method when of Column() spark java Key: SPARK-34867 URL: https://issues.apache.org/jira/browse/SPARK-34867 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-34867) cannot resolve method when of Column() spark java

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308644#comment-17308644 ] Yuming Wang commented on SPARK-34867: - How to reproduce this issue? > cannot resolv

[jira] [Commented] (SPARK-26663) Cannot query a Hive table with subdirectories

2021-03-25 Thread Surendar Thina G (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308648#comment-17308648 ] Surendar Thina G commented on SPARK-26663: -- A similar issue is documented here,

[jira] [Commented] (SPARK-34867) cannot resolve method when of Column() spark java

2021-03-25 Thread unical1988 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308653#comment-17308653 ] unical1988 commented on SPARK-34867: {code:java} firstDF.select(firstDeltaDF.apply("

[jira] [Comment Edited] (SPARK-34867) cannot resolve method when of Column() spark java

2021-03-25 Thread unical1988 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308653#comment-17308653 ] unical1988 edited comment on SPARK-34867 at 3/25/21, 12:30 PM: ---

[jira] [Updated] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-34868: - Description: Support the divide op of year-month interval by numeric types including: # ByteType # Short

[jira] [Created] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Max Gekk (Jira)
Max Gekk created SPARK-34868: Summary: Divide year-month interval by numeric Key: SPARK-34868 URL: https://issues.apache.org/jira/browse/SPARK-34868 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

2021-03-25 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-34869: -- Summary: Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output Key: SPARK-34869 URL: https://issues.apache.org/jira/browse/SPARK-34869

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in https://github.com/apache/spark/pull/31923#issuecomment-

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in https://github.com/apache/spark/pull/31923#issuecomment-8

[jira] [Assigned] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-34869: -- Assignee: Attila Zsolt Piros > Extend k8s "EXTRA LOGS FOR THE FAILED TEST" se

[jira] [Commented] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308679#comment-17308679 ] Attila Zsolt Piros commented on SPARK-34869: I am working on this. > Extend

[jira] [Commented] (SPARK-34833) Apply right-padding correctly for correlated subqueries

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308683#comment-17308683 ] Apache Spark commented on SPARK-34833: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-34833) Apply right-padding correctly for correlated subqueries

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308685#comment-17308685 ] Apache Spark commented on SPARK-34833: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-34786) read parquet uint64 as decimal

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308698#comment-17308698 ] Apache Spark commented on SPARK-34786: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-34786) read parquet uint64 as decimal

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34786: Assignee: Apache Spark > read parquet uint64 as decimal > --

[jira] [Assigned] (SPARK-34786) read parquet uint64 as decimal

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34786: Assignee: (was: Apache Spark) > read parquet uint64 as decimal >

[jira] [Updated] (SPARK-34526) Skip checking glob path in FileStreamSink.hasMetadata

2021-03-25 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-34526: Summary: Skip checking glob path in FileStreamSink.hasMetadata (was: Add a flag to skip checking

[jira] [Updated] (SPARK-34526) Skip checking glob path in FileStreamSink.hasMetadata

2021-03-25 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-34526: Description: Some users may use a very long glob path to read and `isDirectory` may fail when the

[jira] [Created] (SPARK-34870) Jars downloaded with the --packages argument are not added to the classpath for executors.

2021-03-25 Thread Cory Maklin (Jira)
Cory Maklin created SPARK-34870: --- Summary: Jars downloaded with the --packages argument are not added to the classpath for executors. Key: SPARK-34870 URL: https://issues.apache.org/jira/browse/SPARK-34870

[jira] [Assigned] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34868: Assignee: Apache Spark (was: Max Gekk) > Divide year-month interval by numeric > ---

[jira] [Assigned] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34868: Assignee: Max Gekk (was: Apache Spark) > Divide year-month interval by numeric > ---

[jira] [Commented] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308774#comment-17308774 ] Apache Spark commented on SPARK-34868: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-34868) Divide year-month interval by numeric

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308775#comment-17308775 ] Apache Spark commented on SPARK-34868: -- User 'MaxGekk' has created a pull request f

[jira] [Updated] (SPARK-34870) Jars downloaded with the --packages argument are not added to the classpath for executors.

2021-03-25 Thread Cory Maklin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cory Maklin updated SPARK-34870: Description: When Spark is run in local mode, it works as expected. However, when Spark is run in

[jira] [Created] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-34871: --- Summary: Move checkpoint resolving logic to the rule ResolveWriteToStream Key: SPARK-34871 URL: https://issues.apache.org/jira/browse/SPARK-34871 Project: Spark

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in https://github.com/apache/spark/pull/31923#issuecomment-8

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Summary: Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods outp

[jira] [Commented] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308782#comment-17308782 ] Apache Spark commented on SPARK-34869: -- User 'attilapiros' has created a pull reque

[jira] [Assigned] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34869: Assignee: Apache Spark (was: Attila Zsolt Piros) > Extend k8s "EXTRA LOGS FOR THE FAILED

[jira] [Assigned] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34869: Assignee: Attila Zsolt Piros (was: Apache Spark) > Extend k8s "EXTRA LOGS FOR THE FAILED

[jira] [Commented] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308783#comment-17308783 ] Apache Spark commented on SPARK-34869: -- User 'attilapiros' has created a pull reque

[jira] [Assigned] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34871: Assignee: (was: Apache Spark) > Move checkpoint resolving logic to the rule ResolveWr

[jira] [Assigned] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34871: Assignee: Apache Spark > Move checkpoint resolving logic to the rule ResolveWriteToStream

[jira] [Commented] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308787#comment-17308787 ] Apache Spark commented on SPARK-34871: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-33600) Group exception messages in execution/datasources/v2

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33600: --- Assignee: Karen Feng > Group exception messages in execution/datasources/v2 > -

[jira] [Resolved] (SPARK-33600) Group exception messages in execution/datasources/v2

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33600. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31619 [https://gith

[jira] [Resolved] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-25 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-34856. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31954 [https:

[jira] [Created] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-34872: -- Summary: quoteIfNeeded should quote a name which contains non-word characters Key: SPARK-34872 URL: https://issues.apache.org/jira/browse/SPARK-34872 Project: Spa

[jira] [Updated] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34872: --- Description: quoteIfNeeded quotes a name only when it contains . (dots) or ` (backticks) but

[jira] [Assigned] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34872: Assignee: Apache Spark (was: Kousuke Saruta) > quoteIfNeeded should quote a name which c

[jira] [Assigned] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34872: Assignee: Apache Spark (was: Kousuke Saruta) > quoteIfNeeded should quote a name which c

[jira] [Commented] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308799#comment-17308799 ] Apache Spark commented on SPARK-34872: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-34872) quoteIfNeeded should quote a name which contains non-word characters

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34872: Assignee: Kousuke Saruta (was: Apache Spark) > quoteIfNeeded should quote a name which c

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-25 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308811#comment-17308811 ] Michael Chen commented on SPARK-34780: -- Np [~csun]. If the cache isn't materialized

[jira] [Updated] (SPARK-34870) Jars downloaded with the --packages argument are not added to the classpath for executors.

2021-03-25 Thread Cory Maklin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cory Maklin updated SPARK-34870: Description: When Spark is run in local mode, it works as expected. However, when Spark is run in

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308854#comment-17308854 ] Chao Sun commented on SPARK-34780: -- [~mikechen], yes you're right. I'm not sure if this

[jira] [Assigned] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

2021-03-25 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34840: --- Assignee: Chandni Singh > Fix cases of corruption in merged shuffle blocks

[jira] [Resolved] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

2021-03-25 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34840. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue res

[jira] [Commented] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-03-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308872#comment-17308872 ] Dongjoon Hyun commented on SPARK-34859: --- Thank you for creating a new JIRA, [~lxia

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Summary: JDBCRelation columnPartition function improperly determines stride size. Upper bo

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Target Version/s: 3.2.0, 3.1.2, 3.0.3 > JDBCRelation columnPartition function improperly d

[jira] [Commented] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308978#comment-17308978 ] Jason Yarbrough commented on SPARK-34843: - PR created: [Spark 34843 - JDBCRelati

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2021-03-25 Thread Daniel Zhi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308984#comment-17308984 ] Daniel Zhi commented on SPARK-23977: [~ste...@apache.org] How to workaround followin

[jira] [Comment Edited] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2021-03-25 Thread Daniel Zhi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308984#comment-17308984 ] Daniel Zhi edited comment on SPARK-23977 at 3/25/21, 9:40 PM:

[jira] [Comment Edited] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2021-03-25 Thread Daniel Zhi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308984#comment-17308984 ] Daniel Zhi edited comment on SPARK-23977 at 3/25/21, 9:41 PM:

[jira] [Comment Edited] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2021-03-25 Thread Daniel Zhi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308984#comment-17308984 ] Daniel Zhi edited comment on SPARK-23977 at 3/25/21, 9:42 PM:

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

2021-03-25 Thread Chenxi Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309012#comment-17309012 ] Chenxi Zhao commented on SPARK-31754: - I got the same issue. I was using Spark 2.4.4

[jira] [Assigned] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34843: Assignee: Apache Spark > JDBCRelation columnPartition function improperly determines stri

[jira] [Assigned] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34843: Assignee: (was: Apache Spark) > JDBCRelation columnPartition function improperly dete

[jira] [Comment Edited] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

2021-03-25 Thread Chenxi Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309012#comment-17309012 ] Chenxi Zhao edited comment on SPARK-31754 at 3/26/21, 12:08 AM: --

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34843: - Target Version/s: (was: 3.2.0, 3.1.2, 3.0.3) > JDBCRelation columnPartition function i

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed do to stride alignment.

2021-03-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34843: - Affects Version/s: 3.0.3 3.2.0 3.12 > JDBC

[jira] [Resolved] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34871. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31963 [https://gi

[jira] [Assigned] (SPARK-34871) Move checkpoint resolving logic to the rule ResolveWriteToStream

2021-03-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34871: Assignee: Yuanjian Li > Move checkpoint resolving logic to the rule ResolveWriteToStream

[jira] [Assigned] (SPARK-34786) read parquet uint64 as decimal

2021-03-25 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-34786: Assignee: Kent Yao > read parquet uint64 as decimal > -- > >

[jira] [Resolved] (SPARK-34786) read parquet uint64 as decimal

2021-03-25 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-34786. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31960 [https://github.com

[jira] [Updated] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-25 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-34849: Description: This is a SPIP for porting [Koalas project|https://github.com/databricks/koalas] to

[jira] [Commented] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309087#comment-17309087 ] Apache Spark commented on SPARK-34638: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34638: Assignee: Apache Spark > Spark SQL reads unnecessary nested fields (another type of pruni

[jira] [Assigned] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34638: Assignee: (was: Apache Spark) > Spark SQL reads unnecessary nested fields (another ty

[jira] [Commented] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

2021-03-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309088#comment-17309088 ] Apache Spark commented on SPARK-34638: -- User 'viirya' has created a pull request fo

[jira] [Updated] (SPARK-34607) NewInstance.resolved should not throw malformed class name error

2021-03-25 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34607: Fix Version/s: 2.4.8 > NewInstance.resolved should not throw malformed class name error >

[jira] [Updated] (SPARK-34542) Upgrade Parquet to 1.12.0

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34542: Description: Parquet-1.12.0 release notes: https://github.com/apache/parquet-mr/blob/apache-parque

[jira] [Resolved] (SPARK-32855) Improve DPP for some join type do not support broadcast filtering side

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32855. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 29726 [https://gith

[jira] [Assigned] (SPARK-32855) Improve DPP for some join type do not support broadcast filtering side

2021-03-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32855: --- Assignee: Yuming Wang > Improve DPP for some join type do not support broadcast filtering s

[jira] [Assigned] (SPARK-34816) Support for Parquet unsigned LogicalTypes

2021-03-25 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-34816: Assignee: Kent Yao > Support for Parquet unsigned LogicalTypes >

[jira] [Resolved] (SPARK-34816) Support for Parquet unsigned LogicalTypes

2021-03-25 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-34816. -- Fix Version/s: 3.2.0 Resolution: Fixed > Support for Parquet unsigned LogicalTypes > --

  1   2   >