[jira] [Resolved] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28216. - Resolution: Fixed > Add calculate local directory size to SQLTestUtils >

[jira] [Assigned] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-28216: --- Assignee: Yuming Wang > Add calculate local directory size to SQLTestUtils >

[jira] [Updated] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28216: Affects Version/s: (was: 3.1.0) 3.0.0 > Add calculate local directory

[jira] [Updated] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28216: Fix Version/s: 3.0.0 > Add calculate local directory size to SQLTestUtils >

[jira] [Resolved] (SPARK-29868) Add a benchmark for Adaptive Execution

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-29868. - Resolution: Won't Fix > Add a benchmark for Adaptive Execution >

[jira] [Resolved] (SPARK-31288) Remove `getRawTable` in `alterPartitions`

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-31288. - Resolution: Won't Fix > Remove `getRawTable` in `alterPartitions` >

[jira] [Updated] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-14922: Target Version/s: 3.1.0 (was: 3.0.0) > Alter Table Drop Partition Using Predicate-based

[jira] [Assigned] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32722: Assignee: Apache Spark > Update document type conversion for Pandas UDFs (pyarrow 1.0.1,

[jira] [Assigned] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32722: Assignee: (was: Apache Spark) > Update document type conversion for Pandas UDFs

[jira] [Commented] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186259#comment-17186259 ] Apache Spark commented on SPARK-32722: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32722: - Description: Update the chart generated by SPARK-25798. > Update document type conversion for

[jira] [Created] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32722: Summary: Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7) Key: SPARK-32722 URL: https://issues.apache.org/jira/browse/SPARK-32722

[jira] [Commented] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186229#comment-17186229 ] Apache Spark commented on SPARK-28612: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Commented] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186228#comment-17186228 ] Apache Spark commented on SPARK-28612: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32708: - Affects Version/s: 2.4.7 > Query optimization fails to reuse exchange with DataSourceV2

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32708: - Affects Version/s: (was: 2.4.7) (was: 2.4.5)

[jira] [Commented] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186209#comment-17186209 ] Apache Spark commented on SPARK-32721: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32721: Assignee: Apache Spark > Simplify if clauses with null and boolean >

[jira] [Commented] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186206#comment-17186206 ] Apache Spark commented on SPARK-32721: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32721: Assignee: (was: Apache Spark) > Simplify if clauses with null and boolean >

[jira] [Comment Edited] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186202#comment-17186202 ] Patrick Grandjean edited comment on SPARK-32716 at 8/28/20, 2:02 AM: -

[jira] [Commented] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186202#comment-17186202 ] Patrick Grandjean commented on SPARK-32716: --- Yes, but not something that provides both the

[jira] [Commented] (SPARK-31561) Add QUALIFY Clause

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186199#comment-17186199 ] Yuming Wang commented on SPARK-31561: - Note that we should forbid the following case: {code:sql}

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Created] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
Chao Sun created SPARK-32721: Summary: Simplify if clauses with null and boolean Key: SPARK-32721 URL: https://issues.apache.org/jira/browse/SPARK-32721 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Commented] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186196#comment-17186196 ] Hyukjin Kwon commented on SPARK-32720: -- I can't reproduce it by doing below: {code}

[jira] [Resolved] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32693. -- Fix Version/s: 3.1.0 Assignee: L. C. Hsieh Resolution: Fixed Resolved

[jira] [Resolved] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32716. -- Resolution: Won't Fix > Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD >

[jira] [Commented] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186192#comment-17186192 ] Hyukjin Kwon commented on SPARK-32716: -- RDD APIs are currently rather in a maintenance mode so

[jira] [Commented] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186191#comment-17186191 ] Hyukjin Kwon commented on SPARK-32720: -- [~markus.notti] can you make the reproducer self-contained?

[jira] [Commented] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186190#comment-17186190 ] Apache Spark commented on SPARK-32629: -- User 'c21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32629: Assignee: (was: Apache Spark) > Record metrics of extra BitSet/HashSet in full outer

[jira] [Assigned] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32629: Assignee: Apache Spark > Record metrics of extra BitSet/HashSet in full outer shuffled

[jira] [Commented] (SPARK-24994) When the data type of the field is converted to other types, it can also support pushdown to parquet

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186173#comment-17186173 ] Apache Spark commented on SPARK-24994: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32708: Assignee: (was: Apache Spark) > Query optimization fails to reuse exchange with

[jira] [Commented] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186158#comment-17186158 ] Apache Spark commented on SPARK-32708: -- User 'mingjialiu' has created a pull request for this

[jira] [Assigned] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32708: Assignee: Apache Spark > Query optimization fails to reuse exchange with DataSourceV2 >

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a

[jira] [Created] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
Markus created SPARK-32720: -- Summary: Spark 3 Fails to Cast DateType to StringType when comparing result of Max Key: SPARK-32720 URL: https://issues.apache.org/jira/browse/SPARK-32720 Project: Spark

[jira] [Commented] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186115#comment-17186115 ] Apache Spark commented on SPARK-32719: -- User 'Fokko' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32719: Assignee: Apache Spark > Add Flake8 check for missing imports >

[jira] [Commented] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186114#comment-17186114 ] Apache Spark commented on SPARK-32719: -- User 'Fokko' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32719: Assignee: (was: Apache Spark) > Add Flake8 check for missing imports >

[jira] [Created] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Fokko Driesprong (Jira)
Fokko Driesprong created SPARK-32719: Summary: Add Flake8 check for missing imports Key: SPARK-32719 URL: https://issues.apache.org/jira/browse/SPARK-32719 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Description: Update (2020 by Cheng Su): We use this JIRA to track progress for Hive bucketing write

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186023#comment-17186023 ] Cheng Su commented on SPARK-19256: -- [~viirya] - > Do you mean Hive ORC/Parquet bucketed table not

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32713: - Assignee: Dongjoon Hyun > Support execId placeholder in executor PVC conf >

[jira] [Resolved] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32713. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29557

[jira] [Commented] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185936#comment-17185936 ] Apache Spark commented on SPARK-32718: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32718: Assignee: Wenchen Fan (was: Apache Spark) > remove unnecessary keywords for interval

[jira] [Commented] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185934#comment-17185934 ] Apache Spark commented on SPARK-32718: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32718: Assignee: Apache Spark (was: Wenchen Fan) > remove unnecessary keywords for interval

[jira] [Created] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-32718: --- Summary: remove unnecessary keywords for interval units Key: SPARK-32718 URL: https://issues.apache.org/jira/browse/SPARK-32718 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185919#comment-17185919 ] Apache Spark commented on SPARK-31511: -- User 'hvanhovell' has created a pull request for this

[jira] [Commented] (SPARK-32636) AsyncEventQueue: Exception scala.Some cannot be cast to java.lang.String

2020-08-27 Thread Muhammad Abubakar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185866#comment-17185866 ] Muhammad Abubakar commented on SPARK-32636: --- Hi [~hyukjin.kwon], Thanks for looking into it. 

[jira] [Assigned] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-32701: Assignee: Waleed Fateem > mapreduce.fileoutputcommitter.algorithm.version default

[jira] [Resolved] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-32701. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-30654) Update Docs Bootstrap to 4.4.1

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30654: - Fix Version/s: 3.1.0 > Update Docs Bootstrap to 4.4.1 > -- > >

[jira] [Commented] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185838#comment-17185838 ] Apache Spark commented on SPARK-32717: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185837#comment-17185837 ] Apache Spark commented on SPARK-32717: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32717: Assignee: Apache Spark > Add a AQEOptimizer for AdaptiveSparkPlanExec >

[jira] [Assigned] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32717: Assignee: (was: Apache Spark) > Add a AQEOptimizer for AdaptiveSparkPlanExec >

[jira] [Created] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread wuyi (Jira)
wuyi created SPARK-32717: Summary: Add a AQEOptimizer for AdaptiveSparkPlanExec Key: SPARK-32717 URL: https://issues.apache.org/jira/browse/SPARK-32717 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185825#comment-17185825 ] Apache Spark commented on SPARK-32715: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32715: Assignee: Apache Spark > Broadcast block pieces may memory leak >

[jira] [Commented] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185824#comment-17185824 ] Apache Spark commented on SPARK-32715: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32715: Assignee: (was: Apache Spark) > Broadcast block pieces may memory leak >

[jira] [Created] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
Patrick Grandjean created SPARK-32716: - Summary: Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD Key: SPARK-32716 URL: https://issues.apache.org/jira/browse/SPARK-32716 Project:

[jira] [Updated] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32715: --- Description: We use Spark thrift-server as a long-running service. A bad query submitted a heavy

[jira] [Created] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32715: -- Summary: Broadcast block pieces may memory leak Key: SPARK-32715 URL: https://issues.apache.org/jira/browse/SPARK-32715 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32681: - Description: https://github.com/zero323/pyspark-stubs demonstrates a lot of benefits to

[jira] [Updated] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32714: - Description: Port https://github.com/zero323/pyspark-stubs into PySpark. This was being

[jira] [Updated] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32681: - Description: We're discussing about porting https://github.com/zero323/pyspark-stubs in the

[jira] [Updated] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32714: - Description: Port https://github.com/zero323/pyspark-stubs > Port pyspark-stubs >

[jira] [Assigned] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32681: Assignee: Maciej Szymkiewicz > PySpark type hints support > -- >

[jira] [Created] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32714: Summary: Port pyspark-stubs Key: SPARK-32714 URL: https://issues.apache.org/jira/browse/SPARK-32714 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185780#comment-17185780 ] Apache Spark commented on SPARK-32713: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32713: Assignee: (was: Apache Spark) > Support execId placeholder in executor PVC conf >

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32713: Assignee: Apache Spark > Support execId placeholder in executor PVC conf >

[jira] [Commented] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185778#comment-17185778 ] Apache Spark commented on SPARK-32713: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-32713: - Summary: Support execId placeholder in executor PVC conf Key: SPARK-32713 URL: https://issues.apache.org/jira/browse/SPARK-32713 Project: Spark Issue

[jira] [Commented] (SPARK-27951) ANSI SQL: NTH_VALUE function

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185691#comment-17185691 ] Apache Spark commented on SPARK-27951: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-27951) ANSI SQL: NTH_VALUE function

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185690#comment-17185690 ] Apache Spark commented on SPARK-27951: -- User 'beliefer' has created a pull request for this issue:

[jira] [Resolved] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32696. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29539

[jira] [Assigned] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32696: --- Assignee: Kent Yao > Get columns operation should handle interval column properly >

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185649#comment-17185649 ] L. C. Hsieh commented on SPARK-19256: - > Hive on Tez: support zero and multiple files per bucket >

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185648#comment-17185648 ] L. C. Hsieh commented on SPARK-19256: - > Do not allow for writing Hive non-ORC/Parquet bucketed

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

2020-08-27 Thread S (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185643#comment-17185643 ] S commented on SPARK-19335: --- Efficient UPSERT using Spark DataFrame - is it available now? If yes, how can I

[jira] [Resolved] (SPARK-32705) EmptyHashedRelation$; no valid constructor

2020-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32705. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29547

[jira] [Assigned] (SPARK-32705) EmptyHashedRelation$; no valid constructor

2020-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32705: --- Assignee: Leanken.Lin > EmptyHashedRelation$; no valid constructor >

[jira] [Updated] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Affects Version/s: 2.2.0 2.3.0 2.4.0

[jira] [Updated] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Summary: Hive bucketing write support (was: Hive bucketing support) > Hive bucketing write support >

[jira] [Created] (SPARK-32712) Support writing Hive non-ORC/Parquet bucketed table

2020-08-27 Thread Cheng Su (Jira)
Cheng Su created SPARK-32712: Summary: Support writing Hive non-ORC/Parquet bucketed table Key: SPARK-32712 URL: https://issues.apache.org/jira/browse/SPARK-32712 Project: Spark Issue Type:

[jira] [Created] (SPARK-32711) Write Hive ORC/Parquet bucketed table with hive murmur3hash (for Hive 3)

2020-08-27 Thread Cheng Su (Jira)
Cheng Su created SPARK-32711: Summary: Write Hive ORC/Parquet bucketed table with hive murmur3hash (for Hive 3) Key: SPARK-32711 URL: https://issues.apache.org/jira/browse/SPARK-32711 Project: Spark

[jira] [Assigned] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32693: Assignee: (was: Apache Spark) > Compare two dataframes with same schema except

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185621#comment-17185621 ] Apache Spark commented on SPARK-32693: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32693: Assignee: Apache Spark > Compare two dataframes with same schema except nullable

  1   2   >