[jira] [Commented] (SPARK-9853) Optimize shuffle fetch of contiguous partition IDs

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258896#comment-16258896 ] Apache Spark commented on SPARK-9853: - User 'yucai' has created a pull request for this issue:

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

2017-11-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258894#comment-16258894 ] Maciej BryƄski commented on SPARK-16996: [~ste...@apache.org] I didn't replace spark-hive.jar but

[jira] [Commented] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258886#comment-16258886 ] Apache Spark commented on SPARK-22541: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22541: Assignee: Apache Spark > Dataframes: applying multiple filters one after another using

[jira] [Assigned] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22541: Assignee: (was: Apache Spark) > Dataframes: applying multiple filters one after

[jira] [Assigned] (SPARK-22559) history server: handle exception on opening corrupted listing.ldb

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22559: Assignee: (was: Apache Spark) > history server: handle exception on opening corrupted

[jira] [Commented] (SPARK-22559) history server: handle exception on opening corrupted listing.ldb

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258881#comment-16258881 ] Apache Spark commented on SPARK-22559: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-22559) history server: handle exception on opening corrupted listing.ldb

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22559: Assignee: Apache Spark > history server: handle exception on opening corrupted

[jira] [Comment Edited] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258873#comment-16258873 ] Liang-Chi Hsieh edited comment on SPARK-22541 at 11/20/17 7:14 AM: ---

[jira] [Created] (SPARK-22559) history server: handle exception on opening corrupted listing.ldb

2017-11-19 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-22559: -- Summary: history server: handle exception on opening corrupted listing.ldb Key: SPARK-22559 URL: https://issues.apache.org/jira/browse/SPARK-22559 Project: Spark

[jira] [Commented] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258873#comment-16258873 ] Liang-Chi Hsieh commented on SPARK-22541: - Similar to the case of using python udfs with

[jira] [Comment Edited] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258868#comment-16258868 ] Liang-Chi Hsieh edited comment on SPARK-22541 at 11/20/17 7:01 AM: ---

[jira] [Commented] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258868#comment-16258868 ] Liang-Chi Hsieh commented on SPARK-22541: - Sorry, my previous reply is not completely correct.

[jira] [Created] (SPARK-22558) SparkHiveDynamicPartition fails when trying to write data from kafka to hive using spark streaming

2017-11-19 Thread KhajaAsmath Mohammed (JIRA)
KhajaAsmath Mohammed created SPARK-22558: Summary: SparkHiveDynamicPartition fails when trying to write data from kafka to hive using spark streaming Key: SPARK-22558 URL:

[jira] [Resolved] (SPARK-22554) Add a config to control if PySpark should use daemon or not

2017-11-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22554. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19782

[jira] [Assigned] (SPARK-22554) Add a config to control if PySpark should use daemon or not

2017-11-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22554: Assignee: Hyukjin Kwon > Add a config to control if PySpark should use daemon or not >

[jira] [Resolved] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22557. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19784

[jira] [Assigned] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22557: Assignee: Dongjoon Hyun > Use ThreadSignaler explicitly > - >

[jira] [Commented] (SPARK-22556) WrappedArray with Explode Function create WrappedArray with 1 object.

2017-11-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258697#comment-16258697 ] Sean Owen commented on SPARK-22556: --- Are you saying the behavior is incorrect or undesirable? if it's

[jira] [Commented] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258690#comment-16258690 ] Apache Spark commented on SPARK-22557: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22557: Assignee: (was: Apache Spark) > Use ThreadSignaler explicitly >

[jira] [Assigned] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22557: Assignee: Apache Spark > Use ThreadSignaler explicitly > - >

[jira] [Created] (SPARK-22557) Use ThreadSignaler explicitly

2017-11-19 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-22557: - Summary: Use ThreadSignaler explicitly Key: SPARK-22557 URL: https://issues.apache.org/jira/browse/SPARK-22557 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-22556) WrappedArray with Explode Function create WrappedArray with 1 object.

2017-11-19 Thread Thiago Rodrigues Baldim (JIRA)
Thiago Rodrigues Baldim created SPARK-22556: --- Summary: WrappedArray with Explode Function create WrappedArray with 1 object. Key: SPARK-22556 URL: https://issues.apache.org/jira/browse/SPARK-22556

[jira] [Updated] (SPARK-20201) Flaky Test: org.apache.spark.sql.catalyst.expressions.OrderingSuite

2017-11-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-20201: - Target Version/s: 2.2.1 > Flaky Test: org.apache.spark.sql.catalyst.expressions.OrderingSuite >

[jira] [Updated] (SPARK-22543) fix java 64kb compile error for deeply nested expressions

2017-11-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22543: - Target Version/s: 2.2.1, 2.3.0 > fix java 64kb compile error for deeply nested expressions >

[jira] [Updated] (SPARK-22495) Fix setup of SPARK_HOME variable on Windows

2017-11-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22495: - Target Version/s: 2.2.1, 2.3.0 > Fix setup of SPARK_HOME variable on Windows >

[jira] [Commented] (SPARK-21322) support histogram in filter cardinality estimation

2017-11-19 Thread Ron Hu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258601#comment-16258601 ] Ron Hu commented on SPARK-21322: Pull request 19357 was created while there were several dependencies

[jira] [Commented] (SPARK-21322) support histogram in filter cardinality estimation

2017-11-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258596#comment-16258596 ] Apache Spark commented on SPARK-21322: -- User 'ron8hu' has created a pull request for this issue:

[jira] [Created] (SPARK-22555) Possibly incorrect scaling of L2 regularization strength in LinearRegression

2017-11-19 Thread Andrew Crosby (JIRA)
Andrew Crosby created SPARK-22555: - Summary: Possibly incorrect scaling of L2 regularization strength in LinearRegression Key: SPARK-22555 URL: https://issues.apache.org/jira/browse/SPARK-22555

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258487#comment-16258487 ] Sean Owen commented on SPARK-19476: --- Ok. These limitations are from your app though (no batching, high

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Gal Topper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258485#comment-16258485 ] Gal Topper commented on SPARK-19476: The DB supports concurrent requests, but not batching. Meaning

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258482#comment-16258482 ] Sean Owen commented on SPARK-19476: --- But if the DB doesn't like more than 1 concurrent request how do

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-19 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258476#comment-16258476 ] Mark Petruska commented on SPARK-22393: --- Trace of the 2.11 version: {code} ... parse(" class

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Gal Topper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258472#comment-16258472 ] Gal Topper commented on SPARK-19476: > why not more partitions? Because the overhead of 1 slot (or

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-19 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258469#comment-16258469 ] Mark Petruska commented on SPARK-22393: --- The difference between scala repls 2.11 and 2.12 is seen

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-19 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258468#comment-16258468 ] Mark Petruska commented on SPARK-22393: --- With the 2.12 build: {code} import

[jira] [Resolved] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19476. --- Resolution: Not A Problem > Running threads in Spark DataFrame foreachPartition() causes >

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-11-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258467#comment-16258467 ] Sean Owen commented on SPARK-19476: --- Then it's just back to the question: why not more partitions? Why

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-19 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258441#comment-16258441 ] Mark Petruska commented on SPARK-22393: --- Tested with spark-shell build 2.11: {code} import