[jira] [Created] (SPARK-28882) Memory leak when stopping spark session

2019-08-26 Thread Jira
Łukasz Pińkowski created SPARK-28882: Summary: Memory leak when stopping spark session Key: SPARK-28882 URL: https://issues.apache.org/jira/browse/SPARK-28882 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-28667) Support the V2SessionCatalog in insertInto

2019-08-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28667. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25507

[jira] [Assigned] (SPARK-28667) Support the V2SessionCatalog in insertInto

2019-08-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28667: --- Assignee: Burak Yavuz > Support the V2SessionCatalog in insertInto >

[jira] [Resolved] (SPARK-27039) toPandas with Arrow swallows maxResultSize errors

2019-08-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27039. -- Resolution: Duplicate > toPandas with Arrow swallows maxResultSize errors >

[jira] [Reopened] (SPARK-27039) toPandas with Arrow swallows maxResultSize errors

2019-08-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27039: -- > toPandas with Arrow swallows maxResultSize errors >

[jira] [Updated] (SPARK-28881) toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize`

2019-08-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28881: - Fix Version/s: (was: 2.4.4) > toPandas with Arrow returns an empty DataFrame when the

[jira] [Created] (SPARK-28881) toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize`

2019-08-26 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-28881: Summary: toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize` Key: SPARK-28881 URL:

[jira] [Comment Edited] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-26 Thread Franck Tago (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916334#comment-16916334 ] Franck Tago edited comment on SPARK-23519 at 8/27/19 4:10 AM: -- [~viirya]

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-26 Thread Franck Tago (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916334#comment-16916334 ] Franck Tago commented on SPARK-23519: - [~viirya] My mistake , i tested it with Oracle and MySql . I

[jira] [Assigned] (SPARK-22955) Error generating jobs when Stopping JobGenerator gracefully

2019-08-26 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22955: - Assignee: Nikita Gorbachevski > Error generating jobs when Stopping JobGenerator gracefully >

[jira] [Created] (SPARK-28880) ANSI SQL: Bracketed comments

2019-08-26 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-28880: --- Summary: ANSI SQL: Bracketed comments Key: SPARK-28880 URL: https://issues.apache.org/jira/browse/SPARK-28880 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-28879) Kubernetes node selector should be configurable

2019-08-26 Thread Franco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franco updated SPARK-28879: --- Description: Similar to SPARK-25220.   Having to create pod templates is not an answer because: a) It's

[jira] [Created] (SPARK-28879) Kubernetes node selector should be configurable

2019-08-26 Thread Franco (Jira)
Franco created SPARK-28879: -- Summary: Kubernetes node selector should be configurable Key: SPARK-28879 URL: https://issues.apache.org/jira/browse/SPARK-28879 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-28877) Investigate/fix JAXB failure running Pyspark tests on JDK 11

2019-08-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916227#comment-16916227 ] Dongjoon Hyun commented on SPARK-28877: --- I'll make a PR soon. > Investigate/fix JAXB failure

[jira] [Commented] (SPARK-28877) Investigate/fix JAXB failure running Pyspark tests on JDK 11

2019-08-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916218#comment-16916218 ] Dongjoon Hyun commented on SPARK-28877: --- Thank you for filing a Jira issue with the detailed

[jira] [Created] (SPARK-28878) DataSourceV2 should not insert extra projection for columnar batches

2019-08-26 Thread Ryan Blue (Jira)
Ryan Blue created SPARK-28878: - Summary: DataSourceV2 should not insert extra projection for columnar batches Key: SPARK-28878 URL: https://issues.apache.org/jira/browse/SPARK-28878 Project: Spark

[jira] [Created] (SPARK-28877) Investigate/fix JAXB failure running Pyspark tests on JDK 11

2019-08-26 Thread Sean Owen (Jira)
Sean Owen created SPARK-28877: - Summary: Investigate/fix JAXB failure running Pyspark tests on JDK 11 Key: SPARK-28877 URL: https://issues.apache.org/jira/browse/SPARK-28877 Project: Spark

[jira] [Commented] (SPARK-23257) Kerberos Support in Kubernetes resource manager (Cluster Mode)

2019-08-26 Thread Rajendran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916147#comment-16916147 ] Rajendran commented on SPARK-23257: --- Is there a plan to backport these changes to Spark 2.4 versions?

[jira] [Commented] (SPARK-20656) Incremental parsing of event logs in SHS

2019-08-26 Thread Zaheen Aziz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916139#comment-16916139 ] Zaheen Aziz commented on SPARK-20656: - [~shahid] Have you started work on this? > Incremental

[jira] [Resolved] (SPARK-28875) Cover Task retry scenario with test in Kafka connector

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28875. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25582

[jira] [Assigned] (SPARK-28875) Cover Task retry scenario with test in Kafka connector

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28875: -- Assignee: Gabor Somogyi > Cover Task retry scenario with test in Kafka connector >

[jira] [Resolved] (SPARK-28701) add java11 support for spark pull request builds

2019-08-26 Thread shane knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-28701. - Resolution: Fixed Issue resolved by pull request 25585

[jira] [Resolved] (SPARK-28679) Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28679. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25403

[jira] [Assigned] (SPARK-28679) Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28679: -- Assignee: Alessandro Bellina > Spark Yarn ResourceRequestHelper shouldn't lookup

[jira] [Commented] (SPARK-28653) Create table using DDL statement should not auto create the destination folder

2019-08-26 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916032#comment-16916032 ] angerszhu commented on SPARK-28653: --- I tried it in Spark SQL, both situation will create path. >

[jira] [Resolved] (SPARK-28607) Don't hold a reference to two partitionLengths arrays

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28607. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25341

[jira] [Assigned] (SPARK-28607) Don't hold a reference to two partitionLengths arrays

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28607: -- Assignee: Matt Cheah > Don't hold a reference to two partitionLengths arrays >

[jira] [Commented] (SPARK-28872) Will Spark SQL suport the auto analyze for table or partitions like hive by seting hive.stats.autogather=true.

2019-08-26 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915966#comment-16915966 ] angerszhu commented on SPARK-28872: --- See conf  `spark.sql.statistics.size.autoUpdate.enabled` > Will

[jira] [Reopened] (SPARK-28701) add java11 support for spark pull request builds

2019-08-26 Thread shane knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp reopened SPARK-28701: - argh, i broke run-tests.py:

[jira] [Commented] (SPARK-28701) add java11 support for spark pull request builds

2019-08-26 Thread shane knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915948#comment-16915948 ] shane knapp commented on SPARK-28701: - i'm currently trying to fix the hadoop-2.7/jdk-11 build...

[jira] [Updated] (SPARK-27395) New format of EXPLAIN command

2019-08-26 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-27395: Summary: New format of EXPLAIN command (was: Improve EXPLAIN command) > New format of EXPLAIN command >

[jira] [Updated] (SPARK-27395) Improve EXPLAIN command

2019-08-26 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-27395: Issue Type: New Feature (was: Umbrella) > Improve EXPLAIN command > --- > >

[jira] [Resolved] (SPARK-28701) add java11 support for spark pull request builds

2019-08-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28701. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25423

[jira] [Resolved] (SPARK-28709) Memory leaks after stopping of StreamingContext

2019-08-26 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28709. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25439

[jira] [Assigned] (SPARK-28709) Memory leaks after stopping of StreamingContext

2019-08-26 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28709: - Assignee: Nikita Gorbachevski > Memory leaks after stopping of StreamingContext >

[jira] [Updated] (SPARK-28876) fallBackToHdfs should not support Hive partitioned table

2019-08-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28876: Description: The current implementation is incorrect for external partitions and it is expensive

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-26 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915809#comment-16915809 ] Liang-Chi Hsieh commented on SPARK-23519: - I test with Hive 2.1. It doesn't support duplicate

[jira] [Resolved] (SPARK-28527) Build a Test Framework for Thriftserver

2019-08-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28527. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25567

[jira] [Updated] (SPARK-28876) fallBackToHdfs should not support Hive partitioned table

2019-08-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28876: Summary: fallBackToHdfs should not support Hive partitioned table (was: fallBackToHdfs should

[jira] [Created] (SPARK-28876) fallBackToHdfs should not support Hive non-partitioned table

2019-08-26 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-28876: --- Summary: fallBackToHdfs should not support Hive non-partitioned table Key: SPARK-28876 URL: https://issues.apache.org/jira/browse/SPARK-28876 Project: Spark

[jira] [Updated] (SPARK-28875) Cover Task retry scenario with test in Kafka connector

2019-08-26 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28875: -- Summary: Cover Task retry scenario with test in Kafka connector (was: Cover Task retry

[jira] [Resolved] (SPARK-27395) Improve EXPLAIN command

2019-08-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27395. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 24759

[jira] [Assigned] (SPARK-27395) Improve EXPLAIN command

2019-08-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27395: --- Assignee: Dilip Biswal > Improve EXPLAIN command > --- > >

[jira] [Created] (SPARK-28875) Cover Task retry scenario with test

2019-08-26 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-28875: - Summary: Cover Task retry scenario with test Key: SPARK-28875 URL: https://issues.apache.org/jira/browse/SPARK-28875 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

2019-08-26 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Description: In Spark version 2.4 and earlier, when inserting into a table, Spark will cast

[jira] [Updated] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

2019-08-26 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Description: In Spark version 2.4 and earlier, when inserting into a table, Spark will cast

[jira] [Commented] (SPARK-28873) [UDF]show functions behaves different in hive and spark

2019-08-26 Thread Shivu Sondur (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915653#comment-16915653 ] Shivu Sondur commented on SPARK-28873: -- [~hyukjin.kwon] , [~dongjoon] Is this change is required? 

[jira] [Updated] (SPARK-28871) Some codes in 'Policy for handling multiple watermarks' does not show friendly

2019-08-26 Thread chaiyongqiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaiyongqiang updated SPARK-28871: -- Priority: Minor (was: Major) > Some codes in 'Policy for handling multiple watermarks' does

[jira] [Updated] (SPARK-28874) Pyspark bug in date_format

2019-08-26 Thread Luis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis updated SPARK-28874: - Description: Pyspark date_format add one years in the last days off year : Example : {code:python} from

[jira] [Created] (SPARK-28874) Pyspark bug in date_format

2019-08-26 Thread Luis (Jira)
Luis created SPARK-28874: Summary: Pyspark bug in date_format Key: SPARK-28874 URL: https://issues.apache.org/jira/browse/SPARK-28874 Project: Spark Issue Type: Bug Components: PySpark

[jira] [Created] (SPARK-28873) [UDF]show functions behaves different in hive and spark

2019-08-26 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-28873: Summary: [UDF]show functions behaves different in hive and spark Key: SPARK-28873 URL: https://issues.apache.org/jira/browse/SPARK-28873 Project:

[jira] [Commented] (SPARK-28873) [UDF]show functions behaves different in hive and spark

2019-08-26 Thread Shivu Sondur (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915593#comment-16915593 ] Shivu Sondur commented on SPARK-28873: -- i will check this issue > [UDF]show functions behaves

[jira] [Resolved] (SPARK-28693) Malformed input or input contains unmappable characters

2019-08-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28693. - Resolution: Cannot Reproduce Can not reproduce this issue on our jenkins. > Malformed input or

[jira] [Created] (SPARK-28872) Will Spark SQL suport the auto analyze for table or partitions like hive by seting hive.stats.autogather=true.

2019-08-26 Thread Shao (Jira)
Shao created SPARK-28872: Summary: Will Spark SQL suport the auto analyze for table or partitions like hive by seting hive.stats.autogather=true. Key: SPARK-28872 URL: https://issues.apache.org/jira/browse/SPARK-28872

[jira] [Resolved] (SPARK-28829) Document SET ROLE ADMIN in SQL Reference

2019-08-26 Thread ABHISHEK KUMAR GUPTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA resolved SPARK-28829. -- Resolution: Invalid Not valid for doc for Spark. > Document SET ROLE ADMIN

[jira] [Commented] (SPARK-28842) Cleanup the formatting/trailing spaces in resource-managers/kubernetes/integration-tests/README.md

2019-08-26 Thread Udbhav Agrawal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915508#comment-16915508 ] Udbhav Agrawal commented on SPARK-28842: [~holdenk] Can i work on this .. > Cleanup the