[jira] [Created] (SPARK-32784) java.lang.NoClassDefFoundError: parquet/hadoop/ParquetOutputFormat

2020-09-02 Thread clock (Jira)
clock created SPARK-32784: - Summary: java.lang.NoClassDefFoundError: parquet/hadoop/ParquetOutputFormat Key: SPARK-32784 URL: https://issues.apache.org/jira/browse/SPARK-32784 Project: Spark Issue T

[jira] [Updated] (SPARK-32186) User Guide - Debugging

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspar

[jira] [Updated] (SPARK-32186) User Guide - Debugging

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspar

[jira] [Updated] (SPARK-32186) User Guide - Debugging

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspar

[jira] [Created] (SPARK-32783) Development - Testing PySpark

2020-09-02 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32783: Summary: Development - Testing PySpark Key: SPARK-32783 URL: https://issues.apache.org/jira/browse/SPARK-32783 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-32391) Install pydata_sphinx_theme in Jenkins machines

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32391: Assignee: Shane Knapp > Install pydata_sphinx_theme in Jenkins machines > ---

[jira] [Commented] (SPARK-32391) Install pydata_sphinx_theme in Jenkins machines

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189870#comment-17189870 ] Hyukjin Kwon commented on SPARK-32391: -- [~shaneknapp], it's not urgent but when you

[jira] [Assigned] (SPARK-32189) Development - Setting up PyCharm

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32189: Assignee: Haejoon Lee > Development - Setting up PyCharm > --

[jira] [Commented] (SPARK-32189) Development - Setting up PyCharm

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189869#comment-17189869 ] Hyukjin Kwon commented on SPARK-32189: -- Sure, thanks [~itholic]. > Development - S

[jira] [Assigned] (SPARK-32180) Getting Started - Installation

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32180: Assignee: Rohit Mishra > Getting Started - Installation > --

[jira] [Commented] (SPARK-32189) Development - Setting up PyCharm

2020-09-02 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189868#comment-17189868 ] Haejoon Lee commented on SPARK-32189: - I am working on this > Development - Setting

[jira] [Updated] (SPARK-32189) Development - Setting up PyCharm

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32189: - Description: PyCharm is probably one of the most widely used IDE for Python development. We sho

[jira] [Updated] (SPARK-32189) Development - Setting up PyCharm

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32189: - Summary: Development - Setting up PyCharm (was: Development - Development Tools) > Development

[jira] [Commented] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189848#comment-17189848 ] Takeshi Yamamuro commented on SPARK-32778: -- Have you tried the latest releases,

[jira] [Updated] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

2020-09-02 Thread Aman Rastogi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Rastogi updated SPARK-32778: - Description: {code:java} df.write.option("path", "/already/existing/path").mode(SaveMode.Append

[jira] [Commented] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189841#comment-17189841 ] Takeshi Yamamuro commented on SPARK-32747: -- Yay! (Note: I've checked all the re

[jira] [Commented] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189835#comment-17189835 ] Hyukjin Kwon commented on SPARK-32747: -- Thank you Takeshi for correcting it and in

[jira] [Commented] (SPARK-32774) Don't track docs/.jekyll-cache

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189834#comment-17189834 ] Hyukjin Kwon commented on SPARK-32774: -- Thanks [~maropu]! > Don't track docs/.jeky

[jira] [Updated] (SPARK-32771) The example of expressions.Aggregator in Javadoc / Scaladoc is wrong

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32771: - Fix Version/s: (was: 3.0.1) 3.0.2 > The example of expressions.Ag

[jira] [Commented] (SPARK-32782) Refactory StreamingRelationV2 and move it to catalyst

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189826#comment-17189826 ] Apache Spark commented on SPARK-32782: -- User 'xuanyuanking' has created a pull requ

[jira] [Updated] (SPARK-32774) Don't track docs/.jekyll-cache

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32774: - Fix Version/s: (was: 3.0.1) 3.0.2 > Don't track docs/.jekyll-cach

[jira] [Assigned] (SPARK-32782) Refactory StreamingRelationV2 and move it to catalyst

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32782: Assignee: (was: Apache Spark) > Refactory StreamingRelationV2 and move it to catalyst

[jira] [Commented] (SPARK-32782) Refactory StreamingRelationV2 and move it to catalyst

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189829#comment-17189829 ] Apache Spark commented on SPARK-32782: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-32782) Refactory StreamingRelationV2 and move it to catalyst

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32782: Assignee: Apache Spark > Refactory StreamingRelationV2 and move it to catalyst >

[jira] [Commented] (SPARK-32771) The example of expressions.Aggregator in Javadoc / Scaladoc is wrong

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189828#comment-17189828 ] Takeshi Yamamuro commented on SPARK-32771: -- Since v3.0.1 does not include this

[jira] [Commented] (SPARK-32774) Don't track docs/.jekyll-cache

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189827#comment-17189827 ] Takeshi Yamamuro commented on SPARK-32774: -- Since v3.0.1 does not include this

[jira] [Commented] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189825#comment-17189825 ] Takeshi Yamamuro commented on SPARK-32747: -- Since v3.0.1 does not include this

[jira] [Updated] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32747: - Fix Version/s: (was: 3.0.1) 3.0.2 > Deduplicate configuration set

[jira] [Updated] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32693: - Fix Version/s: (was: 3.0.1) 3.0.2 > Compare two dataframes with s

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189823#comment-17189823 ] Takeshi Yamamuro commented on SPARK-32693: -- Unfortunately, v3.0.1 does not incl

[jira] [Created] (SPARK-32782) Refactory StreamingRelationV2 and move it to catalyst

2020-09-02 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-32782: --- Summary: Refactory StreamingRelationV2 and move it to catalyst Key: SPARK-32782 URL: https://issues.apache.org/jira/browse/SPARK-32782 Project: Spark Issue Typ

[jira] [Commented] (SPARK-32739) support prune right for left semi join in DPP

2020-09-02 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189819#comment-17189819 ] Zhenhua Wang commented on SPARK-32739: -- [~maropu]  Added description, thanks for re

[jira] [Updated] (SPARK-32739) support prune right for left semi join in DPP

2020-09-02 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-32739: - Description: Currently in DPP, left semi can only prune left, it should also support prune right

[jira] [Commented] (SPARK-32781) Non-ASCII characters are mistakenly omitted in the middle of intervals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189808#comment-17189808 ] Apache Spark commented on SPARK-32781: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-32781) Non-ASCII characters are mistakenly omitted in the middle of intervals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32781: Assignee: (was: Apache Spark) > Non-ASCII characters are mistakenly omitted in the mi

[jira] [Assigned] (SPARK-32781) Non-ASCII characters are mistakenly omitted in the middle of intervals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32781: Assignee: Apache Spark > Non-ASCII characters are mistakenly omitted in the middle of int

[jira] [Commented] (SPARK-32781) Non-ASCII characters are mistakenly omitted in the middle of intervals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189807#comment-17189807 ] Apache Spark commented on SPARK-32781: -- User 'yaooqinn' has created a pull request

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

2020-09-02 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-32738: - Description: Processing for `ThreadSafeRpcEndpoint` is controlled by 'numActiveThreads' in `Inb

[jira] [Created] (SPARK-32781) Non-ASCII characters are mistakenly omitted in the middle of intervals

2020-09-02 Thread Kent Yao (Jira)
Kent Yao created SPARK-32781: Summary: Non-ASCII characters are mistakenly omitted in the middle of intervals Key: SPARK-32781 URL: https://issues.apache.org/jira/browse/SPARK-32781 Project: Spark

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

2020-09-02 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-32738: - Summary: thread safe endpoints may hang due to fatal error (was: thread safe endpoints may hang

[jira] [Commented] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-09-02 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189785#comment-17189785 ] huangtianhua commented on SPARK-32691: -- [~dongjoon], ok, thanks. And if anyone want

[jira] [Commented] (SPARK-32312) Upgrade Apache Arrow to 1.0.0

2020-09-02 Thread Kazuaki Ishizaki (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189784#comment-17189784 ] Kazuaki Ishizaki commented on SPARK-32312: -- I think that [this|https://github.c

[jira] [Commented] (SPARK-32772) Reduce log messages for spark-sql CLI

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189779#comment-17189779 ] Apache Spark commented on SPARK-32772: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-32772) Reduce log messages for spark-sql CLI

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189778#comment-17189778 ] Apache Spark commented on SPARK-32772: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-32637) SPARK SQL JDBC truncates last value of seconds for datetime2 values for Azure SQL DB

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189771#comment-17189771 ] Takeshi Yamamuro commented on SPARK-32637: -- +1 on the Maxim comment and I'll cl

[jira] [Resolved] (SPARK-32637) SPARK SQL JDBC truncates last value of seconds for datetime2 values for Azure SQL DB

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32637. -- Resolution: Invalid > SPARK SQL JDBC truncates last value of seconds for datetime2 val

[jira] [Updated] (SPARK-32734) RDD actions in DStream.transfrom delays batch submission

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32734: - Labels: (was: pull-request-available) > RDD actions in DStream.transfrom delays batch

[jira] [Updated] (SPARK-32735) RDD actions in DStream.transfrom don't show at batch page

2020-09-02 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32735: - Labels: (was: pull-request-available) > RDD actions in DStream.transfrom don't show at

[jira] [Commented] (SPARK-32735) RDD actions in DStream.transfrom don't show at batch page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189766#comment-17189766 ] Apache Spark commented on SPARK-32735: -- User 'Olwn' has created a pull request for

[jira] [Assigned] (SPARK-32735) RDD actions in DStream.transfrom don't show at batch page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32735: Assignee: Apache Spark > RDD actions in DStream.transfrom don't show at batch page >

[jira] [Commented] (SPARK-32735) RDD actions in DStream.transfrom don't show at batch page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189765#comment-17189765 ] Apache Spark commented on SPARK-32735: -- User 'Olwn' has created a pull request for

[jira] [Assigned] (SPARK-32735) RDD actions in DStream.transfrom don't show at batch page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32735: Assignee: (was: Apache Spark) > RDD actions in DStream.transfrom don't show at batch

[jira] [Commented] (SPARK-32734) RDD actions in DStream.transfrom delays batch submission

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189764#comment-17189764 ] Apache Spark commented on SPARK-32734: -- User 'Olwn' has created a pull request for

[jira] [Assigned] (SPARK-32734) RDD actions in DStream.transfrom delays batch submission

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32734: Assignee: Apache Spark > RDD actions in DStream.transfrom delays batch submission > -

[jira] [Assigned] (SPARK-32734) RDD actions in DStream.transfrom delays batch submission

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32734: Assignee: (was: Apache Spark) > RDD actions in DStream.transfrom delays batch submiss

[jira] [Created] (SPARK-32780) Fill since fields for all the expressions

2020-09-02 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-32780: Summary: Fill since fields for all the expressions Key: SPARK-32780 URL: https://issues.apache.org/jira/browse/SPARK-32780 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32097) Allow reading history log files from multiple directories

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189758#comment-17189758 ] Apache Spark commented on SPARK-32097: -- User 'Gaurangi94' has created a pull reques

[jira] [Assigned] (SPARK-32097) Allow reading history log files from multiple directories

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32097: Assignee: (was: Apache Spark) > Allow reading history log files from multiple directo

[jira] [Assigned] (SPARK-32097) Allow reading history log files from multiple directories

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32097: Assignee: Apache Spark > Allow reading history log files from multiple directories >

[jira] [Created] (SPARK-32779) Spark/Hive3 interaction potentially causes deadlock

2020-09-02 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-32779: - Summary: Spark/Hive3 interaction potentially causes deadlock Key: SPARK-32779 URL: https://issues.apache.org/jira/browse/SPARK-32779 Project: Spark Issue T

[jira] [Commented] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189744#comment-17189744 ] Apache Spark commented on SPARK-32135: -- User 'Gaurangi94' has created a pull reques

[jira] [Assigned] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32135: Assignee: (was: Apache Spark) > Show Spark Driver name on Spark history web page > --

[jira] [Assigned] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32135: Assignee: Apache Spark > Show Spark Driver name on Spark history web page > -

[jira] [Commented] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189743#comment-17189743 ] Apache Spark commented on SPARK-32135: -- User 'Gaurangi94' has created a pull reques

[jira] [Updated] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32764: -- Description: {code:scala} val spark: SparkSession = SparkSession .builder() .mast

[jira] [Commented] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189720#comment-17189720 ] Dongjoon Hyun commented on SPARK-32764: --- cc [~cloud_fan] and [~smilegator] > comp

[jira] [Updated] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32764: -- Affects Version/s: 3.0.1 > compare of -0.0 < 0.0 return true > ---

[jira] [Commented] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189719#comment-17189719 ] Dongjoon Hyun commented on SPARK-32764: --- Thank you for reporting, [~igreenfi]. I a

[jira] [Updated] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32764: -- Target Version/s: 3.0.2 > compare of -0.0 < 0.0 return true >

[jira] [Updated] (SPARK-32764) compare of -0.0 < 0.0 return true

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32764: -- Labels: correctness (was: ) > compare of -0.0 < 0.0 return true > ---

[jira] [Commented] (SPARK-32766) s3a: bucket names with dots cannot be used

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189701#comment-17189701 ] Dongjoon Hyun commented on SPARK-32766: --- Thank you for the pointer, [~ste...@apach

[jira] [Resolved] (SPARK-32772) Reduce log messages for spark-sql CLI

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32772. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29619 [https://

[jira] [Resolved] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32077. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28911 [https://

[jira] [Assigned] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-09-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32077: - Assignee: wuyi > Support host-local shuffle data reading with external shuffle service

[jira] [Comment Edited] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Gaurangi Saxena (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189668#comment-17189668 ] Gaurangi Saxena edited comment on SPARK-32135 at 9/2/20, 7:45 PM:

[jira] [Updated] (SPARK-32097) Allow reading history log files from multiple directories

2020-09-02 Thread Gaurangi Saxena (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurangi Saxena updated SPARK-32097: Description: We would like to configure SparkHistoryServer to display applications from m

[jira] [Updated] (SPARK-32097) Allow reading history log files from multiple directories

2020-09-02 Thread Gaurangi Saxena (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurangi Saxena updated SPARK-32097: Issue Type: Wish (was: Bug) > Allow reading history log files from multiple directories >

[jira] [Commented] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Gaurangi Saxena (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189668#comment-17189668 ] Gaurangi Saxena commented on SPARK-32135: - [~tgraves] Apologies for the late res

[jira] [Updated] (SPARK-32135) Show Spark Driver name on Spark history web page

2020-09-02 Thread Gaurangi Saxena (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurangi Saxena updated SPARK-32135: Attachment: image-2020-09-02-12-37-55-860.png > Show Spark Driver name on Spark history we

[jira] [Created] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

2020-09-02 Thread Aman Rastogi (Jira)
Aman Rastogi created SPARK-32778: Summary: Accidental Data Deletion on calling saveAsTable Key: SPARK-32778 URL: https://issues.apache.org/jira/browse/SPARK-32778 Project: Spark Issue Type: B

[jira] [Commented] (SPARK-28079) CSV fails to detect corrupt record unless "columnNameOfCorruptRecord" is manually added to the schema

2020-09-02 Thread Jeff Harrison (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189634#comment-17189634 ] Jeff Harrison commented on SPARK-28079: --- I would also like something along these l

[jira] [Commented] (SPARK-32746) Not able to run Pandas UDF

2020-09-02 Thread Rahul Bhatia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189589#comment-17189589 ] Rahul Bhatia commented on SPARK-32746: -- Yes, i can run other PySpark codes easily,

[jira] [Commented] (SPARK-32766) s3a: bucket names with dots cannot be used

2020-09-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189279#comment-17189279 ] Steve Loughran commented on SPARK-32766: not going to be fixed in the s3a code,

[jira] [Resolved] (SPARK-31670) Using complex type in Aggregation with cube failed Analysis error

2020-09-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31670. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28490 [https://gith

[jira] [Assigned] (SPARK-31670) Using complex type in Aggregation with cube failed Analysis error

2020-09-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31670: --- Assignee: angerszhu > Using complex type in Aggregation with cube failed Analysis error > -

[jira] [Resolved] (SPARK-32739) support prune right for left semi join in DPP

2020-09-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-32739. - Fix Version/s: 3.1.0 Assignee: Zhenhua Wang Resolution: Fixed Issue resolved by

[jira] [Resolved] (SPARK-32756) Fix CaseInsensitiveMap in Scala 2.13

2020-09-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-32756. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29584 [https://gi

[jira] [Assigned] (SPARK-32756) Fix CaseInsensitiveMap in Scala 2.13

2020-09-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-32756: Assignee: Karol Chmist > Fix CaseInsensitiveMap in Scala 2.13 > -

[jira] [Commented] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189128#comment-17189128 ] Hyukjin Kwon commented on SPARK-32776: -- Sure, thanks [~kabhwan]. > Limit in stream

[jira] [Assigned] (SPARK-32752) Alias breaks for interval typed literals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32752: Assignee: (was: Apache Spark) > Alias breaks for interval typed literals > --

[jira] [Assigned] (SPARK-32752) Alias breaks for interval typed literals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32752: Assignee: Apache Spark > Alias breaks for interval typed literals > -

[jira] [Commented] (SPARK-32752) Alias breaks for interval typed literals

2020-09-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189126#comment-17189126 ] Apache Spark commented on SPARK-32752: -- User 'yaooqinn' has created a pull request

[jira] [Commented] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189125#comment-17189125 ] Jungtaek Lim commented on SPARK-32776: -- It sounds to be safer to mark 3.0.x to 3.0.

[jira] [Updated] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32776: - Fix Version/s: (was: 3.0.1) 3.0.2 > Limit in streaming should not be opti

[jira] [Resolved] (SPARK-32768) Add Parquet Timestamp output configuration to docs

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32768. -- Resolution: Not A Problem > Add Parquet Timestamp output configuration to docs > -

[jira] [Assigned] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32776: Assignee: Liwen Sun > Limit in streaming should not be optimized away by PropagateEmptyRe

[jira] [Resolved] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32776. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed > Limit in streaming sh

[jira] [Commented] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189091#comment-17189091 ] Hyukjin Kwon commented on SPARK-32776: -- Fixed in https://github.com/apache/spark/pu

[jira] [Updated] (SPARK-32317) Parquet file loading with different schema(Decimal(N, P)) in files is not working as expected

2020-09-02 Thread Chen Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhang updated SPARK-32317: --- Labels: (was: easyfix) > Parquet file loading with different schema(Decimal(N, P)) in files is not

[jira] [Updated] (SPARK-32317) Parquet file loading with different schema(Decimal(N, P)) in files is not working as expected

2020-09-02 Thread Chen Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhang updated SPARK-32317: --- Component/s: (was: PySpark) SQL > Parquet file loading with different schema(De

  1   2   >