[jira] [Updated] (SPARK-26056) java api spark streaming spark-avro ui

2018-11-13 Thread wish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wish updated SPARK-26056: - Description: when i use java api spark streaming to read kafka and save avro( databricks spark-avro

[jira] [Created] (SPARK-26056) java api spark streaming spark-avro ui

2018-11-13 Thread wish (JIRA)
wish created SPARK-26056: Summary: java api spark streaming spark-avro ui Key: SPARK-26056 URL: https://issues.apache.org/jira/browse/SPARK-26056 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-26036) Break large tests.py files into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686152#comment-16686152 ] Hyukjin Kwon commented on SPARK-26036: -- I am working on this. > Break large tests.py files into

[jira] [Assigned] (SPARK-26032) Break large sql/tests.py files into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26032: Assignee: Hyukjin Kwon > Break large sql/tests.py files into smaller files >

[jira] [Resolved] (SPARK-26032) Break large sql/tests.py files into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26032. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23021

[jira] [Assigned] (SPARK-26034) Break large mllib/tests.py files into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26034: Assignee: Bryan Cutler > Break large mllib/tests.py files into smaller files >

[jira] [Assigned] (SPARK-26033) Break large ml/tests.py files into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26033: Assignee: Bryan Cutler > Break large ml/tests.py files into smaller files >

[jira] [Commented] (SPARK-26055) InterfaceStability annotations should be retained at runtime

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686112#comment-16686112 ] Apache Spark commented on SPARK-26055: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26055) InterfaceStability annotations should be retained at runtime

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26055: Assignee: Wenchen Fan (was: Apache Spark) > InterfaceStability annotations should be

[jira] [Assigned] (SPARK-26055) InterfaceStability annotations should be retained at runtime

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26055: Assignee: Apache Spark (was: Wenchen Fan) > InterfaceStability annotations should be

[jira] [Created] (SPARK-26055) InterfaceStability annotations should be retained at runtime

2018-11-13 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-26055: --- Summary: InterfaceStability annotations should be retained at runtime Key: SPARK-26055 URL: https://issues.apache.org/jira/browse/SPARK-26055 Project: Spark

[jira] [Updated] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access to tmpDir from $PWD to HDFS

2018-11-13 Thread Greg Senia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated SPARK-25778: --- Attachment: SPARK-25778.4.patch > WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack

[jira] [Updated] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-13 Thread Jaya Krishna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaya Krishna updated SPARK-26054: - Description: When a computed column that rounds the value is added to a data frame, it is

[jira] [Updated] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-13 Thread Jaya Krishna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaya Krishna updated SPARK-26054: - Attachment: sparksql-rounding.png > Creating a computed column applying the spark sql rounding

[jira] [Created] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-13 Thread Jaya Krishna (JIRA)
Jaya Krishna created SPARK-26054: Summary: Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well. Key: SPARK-26054 URL:

[jira] [Resolved] (SPARK-26053) Enhance LikeSimplification

2018-11-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-26053. - Resolution: Not A Problem > Enhance LikeSimplification > -- > >

[jira] [Commented] (SPARK-26053) Enhance LikeSimplification

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686048#comment-16686048 ] Apache Spark commented on SPARK-26053: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26053) Enhance LikeSimplification

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26053: Assignee: Apache Spark > Enhance LikeSimplification > -- > >

[jira] [Assigned] (SPARK-26053) Enhance LikeSimplification

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26053: Assignee: (was: Apache Spark) > Enhance LikeSimplification >

[jira] [Created] (SPARK-26053) Enhance LikeSimplification

2018-11-13 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-26053: --- Summary: Enhance LikeSimplification Key: SPARK-26053 URL: https://issues.apache.org/jira/browse/SPARK-26053 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

2018-11-13 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25994: -- Labels: SPIP (was: ) > SPIP: Property Graphs, Cypher Queries, and Algorithms >

[jira] [Created] (SPARK-26052) Spark should output a _SUCCESS file for every partition correctly written

2018-11-13 Thread Matt Matolcsi (JIRA)
Matt Matolcsi created SPARK-26052: - Summary: Spark should output a _SUCCESS file for every partition correctly written Key: SPARK-26052 URL: https://issues.apache.org/jira/browse/SPARK-26052 Project:

[jira] [Created] (SPARK-26051) Can't create table with column name '22222d'

2018-11-13 Thread Xie Juntao (JIRA)
Xie Juntao created SPARK-26051: -- Summary: Can't create table with column name '2d' Key: SPARK-26051 URL: https://issues.apache.org/jira/browse/SPARK-26051 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685950#comment-16685950 ] Hyukjin Kwon edited comment on SPARK-23831 at 11/14/18 12:54 AM: -

[jira] [Commented] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685950#comment-16685950 ] Hyukjin Kwon commented on SPARK-23831: -- Because it caused a problem that Hive enabled session is

[jira] [Commented] (SPARK-26049) Add InMemoryTable filter benchmark to FilterPushdownBenchmark

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685936#comment-16685936 ] Apache Spark commented on SPARK-26049: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-26050) Implment withColumnExpr method on DataFrame

2018-11-13 Thread Mathew (JIRA)
Mathew created SPARK-26050: -- Summary: Implment withColumnExpr method on DataFrame Key: SPARK-26050 URL: https://issues.apache.org/jira/browse/SPARK-26050 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-26049) Add InMemoryTable filter benchmark to FilterPushdownBenchmark

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26049: Assignee: (was: Apache Spark) > Add InMemoryTable filter benchmark to

[jira] [Commented] (SPARK-26049) Add InMemoryTable filter benchmark to FilterPushdownBenchmark

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685937#comment-16685937 ] Apache Spark commented on SPARK-26049: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26049) Add InMemoryTable filter benchmark to FilterPushdownBenchmark

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26049: Assignee: Apache Spark > Add InMemoryTable filter benchmark to FilterPushdownBenchmark >

[jira] [Created] (SPARK-26049) Add InMemoryTable filter benchmark to FilterPushdownBenchmark

2018-11-13 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-26049: --- Summary: Add InMemoryTable filter benchmark to FilterPushdownBenchmark Key: SPARK-26049 URL: https://issues.apache.org/jira/browse/SPARK-26049 Project: Spark

[jira] [Updated] (SPARK-26048) Flume connector for Spark 2.4 does not exist in Maven repository

2018-11-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26048: --- Target Version/s: 2.4.1 Priority: Blocker (was: Major) > Flume connector for

[jira] [Commented] (SPARK-25598) Remove flume connector in Spark 3

2018-11-13 Thread Aki Tanaka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685914#comment-16685914 ] Aki Tanaka commented on SPARK-25598: Thank you, created SPARK-26048. > Remove flume connector in

[jira] [Created] (SPARK-26048) Flume connector for Spark 2.4 does not exist in Maven repository

2018-11-13 Thread Aki Tanaka (JIRA)
Aki Tanaka created SPARK-26048: -- Summary: Flume connector for Spark 2.4 does not exist in Maven repository Key: SPARK-26048 URL: https://issues.apache.org/jira/browse/SPARK-26048 Project: Spark

[jira] [Updated] (SPARK-26047) Py4JNetworkError (on IPV6): An error occurred while trying to connect to the Java server (127.0.0.1

2018-11-13 Thread Abdul Mateen Mohammed (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdul Mateen Mohammed updated SPARK-26047: -- Summary: Py4JNetworkError (on IPV6): An error occurred while trying to

[jira] [Created] (SPARK-26047) Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1

2018-11-13 Thread Abdul Mateen Mohammed (JIRA)
Abdul Mateen Mohammed created SPARK-26047: - Summary: Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1 Key: SPARK-26047 URL:

[jira] [Commented] (SPARK-25598) Remove flume connector in Spark 3

2018-11-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685841#comment-16685841 ] Marcelo Vanzin commented on SPARK-25598: Hmm, seems that we missed that one. Can you file a

[jira] [Commented] (SPARK-25598) Remove flume connector in Spark 3

2018-11-13 Thread Aki Tanaka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685839#comment-16685839 ] Aki Tanaka commented on SPARK-25598: I understand that flume connector will be removed in Spark 3.

[jira] [Assigned] (SPARK-25960) Support subpath mounting with Kubernetes

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25960: Assignee: (was: Apache Spark) > Support subpath mounting with Kubernetes >

[jira] [Created] (SPARK-26046) Add a way for StreamingQueryManager to remove all listeners

2018-11-13 Thread Mukul Murthy (JIRA)
Mukul Murthy created SPARK-26046: Summary: Add a way for StreamingQueryManager to remove all listeners Key: SPARK-26046 URL: https://issues.apache.org/jira/browse/SPARK-26046 Project: Spark

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685778#comment-16685778 ] Apache Spark commented on SPARK-25960: -- User 'NiharS' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25960) Support subpath mounting with Kubernetes

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25960: Assignee: Apache Spark > Support subpath mounting with Kubernetes >

[jira] [Updated] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

2018-11-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oscar garcía updated SPARK-26045: -- Description: Hello I have been problems with the last spark 2.4 release, the read avro file

[jira] [Updated] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

2018-11-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oscar garcía updated SPARK-26045: -- Description: Hello I have been problems with the last spark 2.4 release, the read avro file

[jira] [Created] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

2018-11-13 Thread JIRA
Oscar garcía created SPARK-26045: - Summary: Error in the spark 2.4 release package with the spark-avro_2.11 depdency Key: SPARK-26045 URL: https://issues.apache.org/jira/browse/SPARK-26045 Project:

[jira] [Updated] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-25924: Attachment: Screen Shot 2018-11-13 at 1.17.50 PM.png > k8s integration tests on jenkins do not

[jira] [Resolved] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-25924. - Resolution: Fixed > k8s integration tests on jenkins do not save log output >

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685748#comment-16685748 ] shane knapp commented on SPARK-25924: - !Screen Shot 2018-11-13 at 1.17.50 PM.png! > k8s integration

[jira] [Assigned] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26024: Assignee: (was: Apache Spark) > Dataset API: repartitionByRange(...) has

[jira] [Assigned] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26024: Assignee: Apache Spark > Dataset API: repartitionByRange(...) has inconsistent behaviour

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685738#comment-16685738 ] Apache Spark commented on SPARK-26024: -- User 'JulienPeloton' has created a pull request for this

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685737#comment-16685737 ] Apache Spark commented on SPARK-26024: -- User 'JulienPeloton' has created a pull request for this

[jira] [Commented] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685719#comment-16685719 ] Apache Spark commented on SPARK-26044: -- User 'shahidki31' has created a pull request for this

[jira] [Commented] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685718#comment-16685718 ] Apache Spark commented on SPARK-26044: -- User 'shahidki31' has created a pull request for this

[jira] [Assigned] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26044: Assignee: Apache Spark > Aggregate Metrics table is not sorted properly based on

[jira] [Assigned] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26044: Assignee: (was: Apache Spark) > Aggregate Metrics table is not sorted properly based

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685713#comment-16685713 ] shane knapp commented on SPARK-25924: - think i got it!  i had the format wrong for the 'archive

[jira] [Updated] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-26044: - Description:   Aggregated Metrics by executor is not properly sorted based on

[jira] [Commented] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685709#comment-16685709 ] shahid commented on SPARK-26044: Thanks. I will raise a PR > Aggregate Metrics table is not sorted

[jira] [Updated] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-26044: - Attachment: image-2018-11-14-01-57-55-948.png > Aggregate Metrics table is not

[jira] [Created] (SPARK-26044) Aggregate Metrics table is not sorted properly based on executor ID

2018-11-13 Thread ABHISHEK KUMAR GUPTA (JIRA)
ABHISHEK KUMAR GUPTA created SPARK-26044: Summary: Aggregate Metrics table is not sorted properly based on executor ID Key: SPARK-26044 URL: https://issues.apache.org/jira/browse/SPARK-26044

[jira] [Updated] (SPARK-25821) Remove SQLContext methods deprecated as of Spark 1.4

2018-11-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25821: -- Labels: release-notes (was: ) Yes certainly > Remove SQLContext methods deprecated as of Spark 1.4

[jira] [Commented] (SPARK-25821) Remove SQLContext methods deprecated as of Spark 1.4

2018-11-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685705#comment-16685705 ] Matt Cheah commented on SPARK-25821: Should we call this out in release notes? It seems pretty

[jira] [Updated] (SPARK-25862) Remove rangeBetween APIs introduced in SPARK-21608

2018-11-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-25862: --- Labels: release-notes (was: ) > Remove rangeBetween APIs introduced in SPARK-21608 >

[jira] [Commented] (SPARK-25862) Remove rangeBetween APIs introduced in SPARK-21608

2018-11-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685704#comment-16685704 ] Matt Cheah commented on SPARK-25862: Adding release-notes label because this seems like a breaking

[jira] [Updated] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-25908: --- Labels: release-notes (was: ) > Remove old deprecated items in Spark 3 >

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685670#comment-16685670 ] shane knapp commented on SPARK-25924: - ok, so that's weird...  i was watching the filesystem on the

[jira] [Comment Edited] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685670#comment-16685670 ] shane knapp edited comment on SPARK-25924 at 11/13/18 7:57 PM: --- ok, so

[jira] [Reopened] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp reopened SPARK-25924: - > k8s integration tests on jenkins do not save log output >

[jira] [Comment Edited] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685670#comment-16685670 ] shane knapp edited comment on SPARK-25924 at 11/13/18 7:55 PM: --- ok, so

[jira] [Created] (SPARK-26043) Make SparkHadoopUtil private to Spark

2018-11-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26043: -- Summary: Make SparkHadoopUtil private to Spark Key: SPARK-26043 URL: https://issues.apache.org/jira/browse/SPARK-26043 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26042: Assignee: Shixiong Zhu (was: Apache Spark) > KafkaContinuousSourceTopicDeletionSuite

[jira] [Commented] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685651#comment-16685651 ] Apache Spark commented on SPARK-26042: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

2018-11-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26042: Assignee: Apache Spark (was: Shixiong Zhu) > KafkaContinuousSourceTopicDeletionSuite

[jira] [Created] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

2018-11-13 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-26042: Summary: KafkaContinuousSourceTopicDeletionSuite may hang forever Key: SPARK-26042 URL: https://issues.apache.org/jira/browse/SPARK-26042 Project: Spark

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685636#comment-16685636 ] shane knapp commented on SPARK-25924: - weird. i'll launch a build and see what's going on... none

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Julien Peloton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685638#comment-16685638 ] Julien Peloton commented on SPARK-26024: [~cloud_fan] I would not advocate to increase the size

[jira] [Commented] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-11-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685629#comment-16685629 ] Michael Armbrust commented on SPARK-23831: -- Why was it reverted? > Add org.apache.derby to

[jira] [Commented] (SPARK-25344) Break large PySpark unittests into smaller files

2018-11-13 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685613#comment-16685613 ] Bryan Cutler commented on SPARK-25344: -- [~hyukjin.kwon] no problem, I can take on ML and MLlib >

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685592#comment-16685592 ] Ruslan Dautkhanov commented on SPARK-26041: --- There are a couple of related jiras that were

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685589#comment-16685589 ] Ruslan Dautkhanov commented on SPARK-26041: --- Issue might be introduced by SPARK-9830  Comment

[jira] [Created] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-13 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-26041: - Summary: catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute Key: SPARK-26041 URL:

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685562#comment-16685562 ] Marcelo Vanzin commented on SPARK-25924: Hey [~shaneknapp], you maked this as fixed but I still

[jira] [Resolved] (SPARK-25942) Aggregate expressions shouldn't be resolved on AppendColumns

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25942. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22944

[jira] [Assigned] (SPARK-25942) Aggregate expressions shouldn't be resolved on AppendColumns

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25942: --- Assignee: Liang-Chi Hsieh > Aggregate expressions shouldn't be resolved on AppendColumns >

[jira] [Assigned] (SPARK-25482) Fast equal can not check reuseSubquery when apply rule about ReuseSubquery

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25482: --- Assignee: Marco Gaido > Fast equal can not check reuseSubquery when apply rule about

[jira] [Resolved] (SPARK-25482) Fast equal can not check reuseSubquery when apply rule about ReuseSubquery

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25482. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22518

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685532#comment-16685532 ] Wenchen Fan commented on SPARK-26024: - range partitioner is usually not very accurate due to

[jira] [Assigned] (SPARK-25947) Reduce memory usage in ShuffleExchangeExec by selecting only the sort columns

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25947: --- Assignee: Shuheng Dai > Reduce memory usage in ShuffleExchangeExec by selecting only the

[jira] [Commented] (SPARK-25344) Break large PySpark unittests into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685494#comment-16685494 ] Hyukjin Kwon commented on SPARK-25344: -- Oh yea. Will do as soon as it gets merged. > Break large

[jira] [Commented] (SPARK-25344) Break large PySpark unittests into smaller files

2018-11-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685485#comment-16685485 ] Imran Rashid commented on SPARK-25344: -- sounds good to me.  it makes sense to break the work up

[jira] [Resolved] (SPARK-25947) Reduce memory usage in ShuffleExchangeExec by selecting only the sort columns

2018-11-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25947. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22961

[jira] [Created] (SPARK-26040) CSV Row delimiters not consistent between platforms

2018-11-13 Thread Heath Abelson (JIRA)
Heath Abelson created SPARK-26040: - Summary: CSV Row delimiters not consistent between platforms Key: SPARK-26040 URL: https://issues.apache.org/jira/browse/SPARK-26040 Project: Spark Issue

[jira] [Commented] (SPARK-26037) ./dev/make-distribution.sh fails for Scala 2.12

2018-11-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685408#comment-16685408 ] Sean Owen commented on SPARK-26037: --- Yes, that's right, -Pscala-2.12 > ./dev/make-distribution.sh

[jira] [Commented] (SPARK-25344) Break large PySpark unittests into smaller files

2018-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685394#comment-16685394 ] Hyukjin Kwon commented on SPARK-25344: -- Hey [~bryanc], once the first try got merged, mind if I ask

[jira] [Commented] (SPARK-26021) -0.0 and 0.0 not treated consistently, doesn't match Hive

2018-11-13 Thread Alon Doron (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685373#comment-16685373 ] Alon Doron commented on SPARK-26021: Since the hashing in UnsafeFixedWidthAggregationMap is done on

[jira] [Commented] (SPARK-14492) Spark SQL 1.6.0 does not work with external Hive metastore version lower than 1.2.0; its not backwards compatible with earlier version

2018-11-13 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685375#comment-16685375 ] Andrew Otto commented on SPARK-14492: - FWIW, I am experiencing the same problem with Spark 2.3.1 and

[jira] [Updated] (SPARK-26039) Reading an empty folder as ORC causes an Analysis Exception

2018-11-13 Thread Abhishek Verma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Verma updated SPARK-26039: --- Description:     {\{val df = spark.read.format("orc").load(orcEmptyFolderPath) }}  

[jira] [Updated] (SPARK-26039) Reading an empty folder as ORC causes an Analysis Exception

2018-11-13 Thread Abhishek Verma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Verma updated SPARK-26039: --- Description:     {\{val df = spark.read.format("orc").load(orcFolderPath) }}  

[jira] [Created] (SPARK-26039) Reading an empty folder as ORC causes an Analysis Exception

2018-11-13 Thread Abhishek Verma (JIRA)
Abhishek Verma created SPARK-26039: -- Summary: Reading an empty folder as ORC causes an Analysis Exception Key: SPARK-26039 URL: https://issues.apache.org/jira/browse/SPARK-26039 Project: Spark

  1   2   >