[jira] [Created] (SPARK-29809) Missing persist in Word2Vec.fit()

2019-11-08 Thread Dong Wang (Jira)
Dong Wang created SPARK-29809: - Summary: Missing persist in Word2Vec.fit() Key: SPARK-29809 URL: https://issues.apache.org/jira/browse/SPARK-29809 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-29808) StopWordsRemover should support multi-cols

2019-11-08 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29808: Summary: StopWordsRemover should support multi-cols Key: SPARK-29808 URL: https://issues.apache.org/jira/browse/SPARK-29808 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29807) Rename "spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled"

2019-11-08 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-29807: Description: The relation between "spark.sql.ansi.enabled" and "spark.sql.dialect" is confusing,

[jira] [Created] (SPARK-29807) Rename "spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled"

2019-11-08 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-29807: --- Summary: Rename "spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled" Key: SPARK-29807 URL: https://issues.apache.org/jira/browse/SPARK-29807 Project:

[jira] [Assigned] (SPARK-28893) support MERGE INTO in the parser and add the corresponding logical plan

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28893: --- Assignee: Xianyin Xin > support MERGE INTO in the parser and add the corresponding logical

[jira] [Resolved] (SPARK-28893) support MERGE INTO in the parser and add the corresponding logical plan

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28893. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26167

[jira] [Resolved] (SPARK-28978) PySpark: Can't pass more than 256 arguments to a UDF

2019-11-08 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-28978. --- Fix Version/s: 3.0.0 Assignee: Bago Amirbekian Resolution: Fixed > PySpark:

[jira] [Created] (SPARK-29806) Using multiline option for a JSON file which is not multiline results in silent truncation of data.

2019-11-08 Thread Dilip Biswal (Jira)
Dilip Biswal created SPARK-29806: Summary: Using multiline option for a JSON file which is not multiline results in silent truncation of data. Key: SPARK-29806 URL:

[jira] [Created] (SPARK-29805) Enable nested schema pruning and pruning on expressions by default

2019-11-08 Thread DB Tsai (Jira)
DB Tsai created SPARK-29805: --- Summary: Enable nested schema pruning and pruning on expressions by default Key: SPARK-29805 URL: https://issues.apache.org/jira/browse/SPARK-29805 Project: Spark

[jira] [Updated] (SPARK-29804) Spark-shell is failing on YARN mode

2019-11-08 Thread Srujan A (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srujan A updated SPARK-29804: - Issue Type: Question (was: Bug) > Spark-shell is failing on YARN mode >

[jira] [Updated] (SPARK-29804) Spark-shell is failing on YARN mode

2019-11-08 Thread Srujan A (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srujan A updated SPARK-29804: - Docs Text: [root@hadoop]# /usr/lib/spark/spark-2.4.4/bin/spark-shell [main] INFO

[jira] [Created] (SPARK-29804) Spark-shell is failing on YARN mode

2019-11-08 Thread Srujan A (Jira)
Srujan A created SPARK-29804: Summary: Spark-shell is failing on YARN mode Key: SPARK-29804 URL: https://issues.apache.org/jira/browse/SPARK-29804 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-29798) Infers bytes as binary type in Python 3 at PySpark

2019-11-08 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-29798. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26432

[jira] [Assigned] (SPARK-29798) Infers bytes as binary type in Python 3 at PySpark

2019-11-08 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-29798: Assignee: Hyukjin Kwon > Infers bytes as binary type in Python 3 at PySpark >

[jira] [Updated] (SPARK-29376) Upgrade Apache Arrow to 0.15.1

2019-11-08 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-29376: - Description: Apache Arrow 0.15.0 was just released see

[jira] [Commented] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970527#comment-16970527 ] Shane Knapp commented on SPARK-29803: - i actually believe that we can do this at any time, as spark

[jira] [Updated] (SPARK-29376) Upgrade Apache Arrow to 0.15.1

2019-11-08 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-29376: - Summary: Upgrade Apache Arrow to 0.15.1 (was: Upgrade Apache Arrow to 0.15.0) > Upgrade

[jira] [Commented] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-08 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970522#comment-16970522 ] Bryan Cutler commented on SPARK-29803: -- This should be done once Python 2 support is dropped

[jira] [Resolved] (SPARK-29673) upgrade jenkins pypy to PyPy3.6 v7.2.0

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-29673. - Resolution: Fixed > upgrade jenkins pypy to PyPy3.6 v7.2.0 >

[jira] [Updated] (SPARK-29672) remove python2 tests and test infra

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Attachment: (was: print_function_list.txt) > remove python2 tests and test infra >

[jira] [Updated] (SPARK-29672) remove python2 tests and test infra

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Description: python 2.7 is EOL jan 1st 2020: [https://github.com/python/devguide/pull/344] it's

[jira] [Created] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-08 Thread Shane Knapp (Jira)
Shane Knapp created SPARK-29803: --- Summary: remove all instances of 'from __future__ import print_function' Key: SPARK-29803 URL: https://issues.apache.org/jira/browse/SPARK-29803 Project: Spark

[jira] [Updated] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29803: Attachment: print_function_list.txt > remove all instances of 'from __future__ import

[jira] [Created] (SPARK-29802) update remaining python scripts in repo to python3 shebang

2019-11-08 Thread Shane Knapp (Jira)
Shane Knapp created SPARK-29802: --- Summary: update remaining python scripts in repo to python3 shebang Key: SPARK-29802 URL: https://issues.apache.org/jira/browse/SPARK-29802 Project: Spark

[jira] [Commented] (SPARK-29673) upgrade jenkins pypy to PyPy3.6 v7.2.0

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970496#comment-16970496 ] Shane Knapp commented on SPARK-29673: - [~hyukjin.kwon] pypy3.6 is available on all jenkins workers. 

[jira] [Updated] (SPARK-29672) remove python2 tests and test infra

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Description: python 2.7 is EOL jan 1st 2020: [https://github.com/python/devguide/pull/344] it's

[jira] [Updated] (SPARK-29672) remove python2 tests and test infra

2019-11-08 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Attachment: print_function_list.txt > remove python2 tests and test infra >

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

2019-11-08 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970451#comment-16970451 ] Felix Kizhakkel Jose commented on SPARK-29764: -- How do I get a help once the priority is

[jira] [Resolved] (SPARK-29790) Add notes about port being required for Kubernetes API URL when set as master

2019-11-08 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29790. Fix Version/s: 3.0.0 2.4.5 Assignee: Emil

[jira] [Resolved] (SPARK-29679) Make interval type camparable and orderable

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29679. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26337

[jira] [Assigned] (SPARK-29679) Make interval type camparable and orderable

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29679: --- Assignee: Kent Yao > Make interval type camparable and orderable >

[jira] [Resolved] (SPARK-29688) Support average with interval type values

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29688. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26347

[jira] [Assigned] (SPARK-29688) Support average with interval type values

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29688: --- Assignee: Kent Yao > Support average with interval type values >

[jira] [Assigned] (SPARK-28477) Rewrite `CASE WHEN cond THEN ifTrue OTHERWISE ifFalse` END into `IF(cond, ifTrue, ifFalse)`

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28477: --- Assignee: David Vrba > Rewrite `CASE WHEN cond THEN ifTrue OTHERWISE ifFalse` END into

[jira] [Resolved] (SPARK-28477) Rewrite `CASE WHEN cond THEN ifTrue OTHERWISE ifFalse` END into `IF(cond, ifTrue, ifFalse)`

2019-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28477. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26294

[jira] [Created] (SPARK-29801) ML models unify toString method

2019-11-08 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29801: Summary: ML models unify toString method Key: SPARK-29801 URL: https://issues.apache.org/jira/browse/SPARK-29801 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-29756) CountVectorizer forget to unpersist intermediate rdd

2019-11-08 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-29756. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26398

[jira] [Assigned] (SPARK-29756) CountVectorizer forget to unpersist intermediate rdd

2019-11-08 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-29756: Assignee: zhengruifeng > CountVectorizer forget to unpersist intermediate rdd >

[jira] [Commented] (SPARK-29800) Plan Exists 's subquery in PlanSubqueries

2019-11-08 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970040#comment-16970040 ] angerszhu commented on SPARK-29800: --- raise pr soon > Plan Exists 's subquery in PlanSubqueries >

[jira] [Created] (SPARK-29800) Plan Exists 's subquery in PlanSubqueries

2019-11-08 Thread angerszhu (Jira)
angerszhu created SPARK-29800: - Summary: Plan Exists 's subquery in PlanSubqueries Key: SPARK-29800 URL: https://issues.apache.org/jira/browse/SPARK-29800 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29408) Support interval literal with negative sign `-`

2019-11-08 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-29408: --- Description: For example: {code} maxim=# select -interval '1 day -1 hour'; ?column?

[jira] [Commented] (SPARK-21453) Cached Kafka consumer may be closed too early

2019-11-08 Thread Julian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969995#comment-16969995 ] Julian commented on SPARK-21453: As an update, the issue "{color:#172b4d}Failed to send SSL Close

[jira] [Comment Edited] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

2019-11-08 Thread Madhusoodan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969977#comment-16969977 ] Madhusoodan edited comment on SPARK-5342 at 11/8/19 9:29 AM: - [~hshreedharan]

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

2019-11-08 Thread Madhusoodan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969977#comment-16969977 ] Madhusoodan commented on SPARK-5342: [~hshreedharan] is it so that only HDFS tokens are renewed? What