[jira] [Created] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread ulysses you (Jira)
ulysses you created SPARK-34852: --- Summary: Close Hive session state should use withHiveState Key: SPARK-34852 URL: https://issues.apache.org/jira/browse/SPARK-34852 Project: Spark Issue Type: I

[jira] [Commented] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307607#comment-17307607 ] Apache Spark commented on SPARK-34852: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34852: Assignee: Apache Spark > Close Hive session state should use withHiveState >

[jira] [Assigned] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34852: Assignee: (was: Apache Spark) > Close Hive session state should use withHiveState > -

[jira] [Commented] (SPARK-34851) Add scope tag in configuration

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307608#comment-17307608 ] Takeshi Yamamuro commented on SPARK-34851: -- Fill the description, please. > Ad

[jira] [Updated] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34849: - Description: This is a SPIP for porting [Koalas project|https://github.com/databricks/koalas] t

[jira] [Updated] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34849: - Description: This is a SPIP for porting [Koalas project|https://github.com/databricks/koalas] t

[jira] [Updated] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34849: - Description: This is a SPIP for porting [Koalas project|https://github.com/databricks/koalas] t

[jira] [Updated] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34849: - Shepherd: Hyukjin Kwon > SPIP: Support pandas API layer on PySpark > ---

[jira] [Created] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Cheng Su (Jira)
Cheng Su created SPARK-34853: Summary: Move partitioning and ordering to common limit trait Key: SPARK-34853 URL: https://issues.apache.org/jira/browse/SPARK-34853 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-34849) SPIP: Support pandas API layer on PySpark

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307610#comment-17307610 ] Hyukjin Kwon commented on SPARK-34849: -- The official SPIP preparation is in progres

[jira] [Commented] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307614#comment-17307614 ] Apache Spark commented on SPARK-34853: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34853: Assignee: (was: Apache Spark) > Move partitioning and ordering to common limit trait

[jira] [Assigned] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34853: Assignee: Apache Spark > Move partitioning and ordering to common limit trait > -

[jira] [Commented] (SPARK-34831) spark2.3 can't add column in carbondata table

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307618#comment-17307618 ] Hyukjin Kwon commented on SPARK-34831: -- It's very likely from the source implementa

[jira] [Resolved] (SPARK-34831) spark2.3 can't add column in carbondata table

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34831. -- Resolution: Not A Problem > spark2.3 can't add column in carbondata table > --

[jira] [Created] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Yijia Cui (Jira)
Yijia Cui created SPARK-34854: - Summary: Report metrics for streaming source through progress reporter with Kafka source use-case Key: SPARK-34854 URL: https://issues.apache.org/jira/browse/SPARK-34854 Pr

[jira] [Resolved] (SPARK-34830) Some UDF calls inside transform are broken

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34830. -- Resolution: Duplicate It's very likely the same issue I believe .. > Some UDF calls inside tr

[jira] [Commented] (SPARK-34825) pyspark.sql.function.lit is treating '1' the same as 1

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307623#comment-17307623 ] Hyukjin Kwon commented on SPARK-34825: -- It treats as a string properly. It just int

[jira] [Resolved] (SPARK-34825) pyspark.sql.function.lit is treating '1' the same as 1

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34825. -- Resolution: Invalid > pyspark.sql.function.lit is treating '1' the same as 1 > ---

[jira] [Updated] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Yijia Cui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yijia Cui updated SPARK-34854: -- Description: Report Metrics for streaming source through progress reporter to users. Add kafka micro

[jira] [Updated] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Yijia Cui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yijia Cui updated SPARK-34854: -- Description: Report Metrics for streaming source through progress reporter to users. Add kafka micro

[jira] [Updated] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Yijia Cui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yijia Cui updated SPARK-34854: -- Description: Report Metrics for streaming source through progress reporter to users. Add kafka micro

[jira] [Resolved] (SPARK-34477) Kryo NPEs when serializing Avro GenericData objects (except GenericRecord)

2021-03-24 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-34477. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31597 [https:

[jira] [Assigned] (SPARK-34477) Kryo NPEs when serializing Avro GenericData objects (except GenericRecord)

2021-03-24 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-34477: -- Assignee: Shardul Mahadik > Kryo NPEs when serializing Avro GenericData objects (exce

[jira] [Commented] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307630#comment-17307630 ] Apache Spark commented on SPARK-34854: -- User 'yijiacui-db' has created a pull reque

[jira] [Assigned] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34854: Assignee: (was: Apache Spark) > Report metrics for streaming source through progress

[jira] [Assigned] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34854: Assignee: Apache Spark > Report metrics for streaming source through progress reporter wi

[jira] [Commented] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307631#comment-17307631 ] Apache Spark commented on SPARK-34854: -- User 'yijiacui-db' has created a pull reque

[jira] [Updated] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-31531: - Attachment: image.png > sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not fo

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Priority: Major (was: Minor) > Allow several properties files > ---

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Affects Version/s: 3.1.1 > Allow several properties files >

[jira] [Updated] (SPARK-34851) Add scope tag in configuration

2021-03-24 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-34851: -- Description: According to discussion at [https://github.com/apache/spark/pull/31598#discussion_r58615

[jira] [Resolved] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34295. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31761 [https://gith

[jira] [Commented] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307644#comment-17307644 ] Arseniy Tashoyan commented on SPARK-34345: -- Typesafe Config seems attractive, a

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Component/s: Kubernetes > Allow several properties files > -

[jira] [Assigned] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34850: Assignee: Apache Spark (was: Max Gekk) > Multiply day-time interval by numeric > ---

[jira] [Assigned] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34850: Assignee: Max Gekk (was: Apache Spark) > Multiply day-time interval by numeric > ---

[jira] [Commented] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307655#comment-17307655 ] Apache Spark commented on SPARK-34850: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

2021-03-24 Thread Yue Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307686#comment-17307686 ] Yue Peng commented on SPARK-34684: -- Why we need to do this? Accessing files from execut

[jira] [Commented] (SPARK-30641) Project Matrix: Linear Models revisit and refactor

2021-03-24 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307690#comment-17307690 ] Weichen Xu commented on SPARK-30641: Good work! > Project Matrix: Linear Models rev

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307697#comment-17307697 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Updated] (SPARK-30641) Project Matrix: Linear Models revisit and refactor

2021-03-24 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30641: - Description: We had been refactoring linear models for a long time, and there still are some wo

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307698#comment-17307698 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307699#comment-17307699 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307700#comment-17307700 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307702#comment-17307702 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

2021-03-24 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307710#comment-17307710 ] Attila Zsolt Piros commented on SPARK-34684: I did not say there is no error

[jira] [Created] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
Li Xian created SPARK-34855: --- Summary: SparkContext - avoid using local lazy val Key: SPARK-34855 URL: https://issues.apache.org/jira/browse/SPARK-34855 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Xian updated SPARK-34855: Attachment: Screen Shot 2021-03-24 at 5.41.22 PM.png > SparkContext - avoid using local lazy val > ---

[jira] [Commented] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307718#comment-17307718 ] Apache Spark commented on SPARK-34855: -- User 'lxian' has created a pull request for

[jira] [Assigned] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34855: Assignee: (was: Apache Spark) > SparkContext - avoid using local lazy val > -

[jira] [Assigned] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34855: Assignee: Apache Spark > SparkContext - avoid using local lazy val >

[jira] [Commented] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307717#comment-17307717 ] Li Xian commented on SPARK-34855: - [https://github.com/apache/spark/pull/31953] I have s

[jira] [Resolved] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34488. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31611 [https://gi

[jira] [Assigned] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34488: Assignee: Ron Hu > Support task Metrics Distributions and executor Metrics Distributions

[jira] [Resolved] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-34853. -- Fix Version/s: 3.2.0 Assignee: Cheng Su Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-34838) consider markdownlint

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34838: - Issue Type: Improvement (was: Bug) > consider markdownlint > - > >

[jira] [Resolved] (SPARK-34801) java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34801. -- Resolution: Not A Problem > java.lang.NoSuchMethodException: > org.apache.hadoop.hive.ql.meta

[jira] [Resolved] (SPARK-34759) run JavaSparkSQLExample failed with Exception.

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34759. -- Resolution: Duplicate > run JavaSparkSQLExample failed with Exception. > -

[jira] [Updated] (SPARK-34757) Spark submit should ignore cache for SNAPSHOT dependencies

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34757: - Priority: Minor (was: Major) > Spark submit should ignore cache for SNAPSHOT dependencies > ---

[jira] [Resolved] (SPARK-34746) Spark dependencies require scala 2.12.12

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34746. -- Resolution: Duplicate Yep, though there are reasons we couldn't use 2.12.12 > Spark dependenc

[jira] [Resolved] (SPARK-34689) Spark Thrift Server: Memory leak for SparkSession objects

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34689. -- Resolution: Duplicate > Spark Thrift Server: Memory leak for SparkSession objects > --

[jira] [Resolved] (SPARK-34630) Add type hints of pyspark.__version__ and pyspark.sql.Column.contains

2021-03-24 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-34630. Fix Version/s: 3.1.2 3.2.0 Assignee: Danny Meijer

[jira] [Assigned] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34822: --- Assignee: Tanel Kiis > Update plan stability golden files even if only explain differs > --

[jira] [Resolved] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34822. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31927 [https://gith

[jira] [Updated] (SPARK-34591) Pyspark undertakes pruning of decision trees and random forests outside the control of the user, leading to undesirable and unexpected outcomes that are challenging to d

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34591: - Priority: Minor (was: Critical) (Not nearly Critical) Yes please open a PR to add the parameter

[jira] [Commented] (SPARK-34569) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307876#comment-17307876 ] Sean R. Owen commented on SPARK-34569: -- This sounds like a Spark version mismatch?

[jira] [Commented] (SPARK-34530) logError for interrupting block migrations is too high

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307878#comment-17307878 ] Sean R. Owen commented on SPARK-34530: -- Is this just about turning log level down?

[jira] [Commented] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307879#comment-17307879 ] Sean R. Owen commented on SPARK-34510: -- Firstly, does this happen on Apache Spark o

[jira] [Resolved] (SPARK-34769) AnsiTypeCoercion: return narrowest convertible type among TypeCollection

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34769. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31859 [https://gith

[jira] [Updated] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34756: Fix Version/s: 3.0.3 > Fix FileScan equality check > --- > >

[jira] [Updated] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-33482: Fix Version/s: 3.0.3 > V2 Datasources that extend FileScan preclude exchange reuse > -

[jira] [Created] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-34856: -- Summary: ANSI mode: Allow casting complex types as string type Key: SPARK-34856 URL: https://issues.apache.org/jira/browse/SPARK-34856 Project: Spark Iss

[jira] [Commented] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307910#comment-17307910 ] Apache Spark commented on SPARK-34856: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34856: Assignee: Apache Spark (was: Gengliang Wang) > ANSI mode: Allow casting complex types as

[jira] [Assigned] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34856: Assignee: Gengliang Wang (was: Apache Spark) > ANSI mode: Allow casting complex types as

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2021-03-24 Thread Xinli Shang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308000#comment-17308000 ] Xinli Shang commented on SPARK-26345: - Yes, it needs some synchronization. I have th

[jira] [Resolved] (SPARK-34797) Refactor Logistic Aggregator - support virtual centering

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34797. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31889 [https://gi

[jira] [Assigned] (SPARK-34797) Refactor Logistic Aggregator - support virtual centering

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34797: Assignee: zhengruifeng (was: Apache Spark) > Refactor Logistic Aggregator - support virt

[jira] [Commented] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308034#comment-17308034 ] Apache Spark commented on SPARK-34829: -- User 'peter-toth' has created a pull reques

[jira] [Assigned] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34829: Assignee: (was: Apache Spark) > transform_values return identical values when it's us

[jira] [Assigned] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34829: Assignee: Apache Spark > transform_values return identical values when it's used with udf

[jira] [Assigned] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34596: --- Assignee: Kris Mok > NewInstance.doGenCode should not throw malformed class name error > --

[jira] [Updated] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34596: Fix Version/s: 2.4.8 > NewInstance.doGenCode should not throw malformed class name error > ---

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Description: I see this in an explain plan {noformat} (12) Filter Input [3]: [c1#2410L, c2#24

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Environment: (was: I see this in an explain plan {noformat} (12) Filter Input [3]: [c1#241

[jira] [Created] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
Tim Armstrong created SPARK-34857: - Summary: AtLeastNNonNulls does not show up correctly in explain Key: SPARK-34857 URL: https://issues.apache.org/jira/browse/SPARK-34857 Project: Spark Issu

[jira] [Commented] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308055#comment-17308055 ] Tim Armstrong commented on SPARK-34857: --- [~LI,Xiao] i would like to take this one

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Priority: Minor (was: Major) > AtLeastNNonNulls does not show up correctly in explain > -

[jira] [Commented] (SPARK-30497) migrate DESCRIBE TABLE to the new framework

2021-03-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308067#comment-17308067 ] Chao Sun commented on SPARK-30497: -- [~cloud_fan] this is resolved right? > migrate DES

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2021-03-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308144#comment-17308144 ] Dongjoon Hyun commented on SPARK-26345: --- Thank you for reporting. Could you file a

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as follo

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as follo

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as follo

[jira] [Resolved] (SPARK-34833) Apply right-padding correctly for correlated subqueries

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-34833. -- Fix Version/s: 3.1.2 3.2.0 Assignee: Takeshi Yamamuro

[jira] [Commented] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308249#comment-17308249 ] Jason Yarbrough commented on SPARK-34844: - As I coded up the unit tests and chec

[jira] [Resolved] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough resolved SPARK-34844. - Resolution: Not A Problem > JDBCRelation columnPartition function includes the first str

[jira] [Comment Edited] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308249#comment-17308249 ] Jason Yarbrough edited comment on SPARK-34844 at 3/25/21, 12:19 AM: --

[jira] [Updated] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34829: - Affects Version/s: 3.2.0 > transform_values return identical values when it's used with

  1   2   >