[jira] [Commented] (SPARK-23114) Spark R 2.3 QA umbrella

2018-01-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328419#comment-16328419 ] Felix Cheung commented on SPARK-23114: -- sure, [~josephkb] > Spark R 2.3 QA umbrella

[jira] [Comment Edited] (SPARK-23115) SparkR 2.3 QA: New R APIs and API docs

2018-01-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328422#comment-16328422 ] Felix Cheung edited comment on SPARK-23115 at 1/17/18 8:01 AM:

[jira] [Commented] (SPARK-23115) SparkR 2.3 QA: New R APIs and API docs

2018-01-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328422#comment-16328422 ] Felix Cheung commented on SPARK-23115: -- did this, and opened this https://issues.ap

[jira] [Resolved] (SPARK-23062) EXCEPT documentation should make it clear that it's EXCEPT DISTINCT

2018-01-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23062. - Resolution: Fixed Assignee: Henry Robinson Fix Version/s: 2.3.0 > EXCEPT documentation sh

[jira] [Created] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
zhaoshijie created SPARK-23125: -- Summary: Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout. Key: SPARK-23125 URL: https://issues.apache.org/jira/browse/SPARK-23125

[jira] [Commented] (SPARK-23118) SparkR 2.3 QA: Programming guide, migration guide, vignettes updates

2018-01-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328430#comment-16328430 ] Felix Cheung commented on SPARK-23118: -- did this and opened SPARK-21616 > SparkR 2.

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more th

[jira] [Assigned] (SPARK-23020) Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23020: Assignee: Apache Spark (was: Marcelo Vanzin) > Flaky Test: org.apache.spark.launcher.Spar

[jira] [Assigned] (SPARK-23020) Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23020: Assignee: Marcelo Vanzin (was: Apache Spark) > Flaky Test: org.apache.spark.launcher.Spar

[jira] [Commented] (SPARK-23020) Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328455#comment-16328455 ] Apache Spark commented on SPARK-23020: -- User 'sameeragarwal' has created a pull requ

[jira] [Updated] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-23020: --- Summary: Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLaun

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more t

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more t

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more t

[jira] [Created] (SPARK-23126) I used the Project operator and modified the source. After compiling successfully, and testing the jars, I got the exception. Maybe the phenomenon is related with implic

2018-01-17 Thread xuetao (JIRA)
xuetao created SPARK-23126: -- Summary: I used the Project operator and modified the source. After compiling successfully, and testing the jars, I got the exception. Maybe the phenomenon is related with implicits Key: SPARK-23126

[jira] [Created] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-17 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23127: -- Summary: Update FeatureHasher user guide for catCols parameter Key: SPARK-23127 URL: https://issues.apache.org/jira/browse/SPARK-23127 Project: Spark Iss

[jira] [Updated] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-17 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23127: --- Description: SPARK-22801 added the {{categoricalCols}} parameter and updated the Scala and Py

[jira] [Created] (SPARK-23128) Introduce QueryStage to improve adaptive execution in Spark SQL

2018-01-17 Thread Carson Wang (JIRA)
Carson Wang created SPARK-23128: --- Summary: Introduce QueryStage to improve adaptive execution in Spark SQL Key: SPARK-23128 URL: https://issues.apache.org/jira/browse/SPARK-23128 Project: Spark

[jira] [Created] (SPARK-23129) Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times

2018-01-17 Thread zhoukang (JIRA)
zhoukang created SPARK-23129: Summary: Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times Key: SPARK-23129 URL: https://issues.apache.org/jira/browse/SP

[jira] [Assigned] (SPARK-23129) Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23129: Assignee: (was: Apache Spark) > Lazy init DiskMapIterator#deserializeStream to reduce

[jira] [Assigned] (SPARK-23129) Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23129: Assignee: Apache Spark > Lazy init DiskMapIterator#deserializeStream to reduce memory usag

[jira] [Commented] (SPARK-23129) Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328543#comment-16328543 ] Apache Spark commented on SPARK-23129: -- User 'caneGuy' has created a pull request fo

[jira] [Created] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
Sean Roberts created SPARK-23130: Summary: Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout) Key: SPARK-23130 URL: https://issues.apache.org/jira/browse/SPARK-23130

[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Roberts updated SPARK-23130: - Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0 * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4

[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Roberts updated SPARK-23130: - Environment: * Spark versions: 1.6.3, 2.1.0, 2.2.0 * Hadoop distributions: HDP 2.5 - 2.6.3.0 *

[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Roberts updated SPARK-23130: - Description: Spark Thrift is not cleaning up /tmp for files & directories named like: /tmp/hive/

[jira] [Commented] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328568#comment-16328568 ] Sean Roberts commented on SPARK-23130: -- * SPARK-15401: Similar report for the "_reso

[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Roberts updated SPARK-23130: - Labels: thrift (was: ) > Spark Thrift does not clean-up temporary files (/tmp/*_resources and >

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more t

[jira] [Commented] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328584#comment-16328584 ] Apache Spark commented on SPARK-23127: -- User 'MLnick' has created a pull request for

[jira] [Assigned] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23127: Assignee: (was: Apache Spark) > Update FeatureHasher user guide for catCols parameter

[jira] [Assigned] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23127: Assignee: Apache Spark > Update FeatureHasher user guide for catCols parameter > -

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

2018-01-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328651#comment-16328651 ] Steve Loughran commented on SPARK-23123: I've never looked at ViewFS internals be

[jira] [Created] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
Peigen created SPARK-23131: -- Summary: Stackoverflow using ML and Kryo serializer Key: SPARK-23131 URL: https://issues.apache.org/jira/browse/SPARK-23131 Project: Spark Issue Type: Bug Comp

[jira] [Updated] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peigen updated SPARK-23131: --- Description: When trying to use GeneralizedLinearRegression model and set SparkConf to use KryoSerializer(Ja

[jira] [Updated] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peigen updated SPARK-23131: --- Environment: (was: When trying to use GeneralizedLinearRegression model and set SparkConf to use KryoSeri

[jira] [Updated] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peigen updated SPARK-23131: --- Priority: Minor (was: Critical) > Stackoverflow using ML and Kryo serializer > -

[jira] [Updated] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-23125: --- Description: I find DirectKafkaInputDStream(kafka010) Offset commit failed when batch time is more t

[jira] [Updated] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23132: - Environment: (was: Seems currently we don't actually run the doctests in \{{ml.image.py}}. It

[jira] [Created] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-23132: Summary: Run ml.image doctests in tests Key: SPARK-23132 URL: https://issues.apache.org/jira/browse/SPARK-23132 Project: Spark Issue Type: Test Com

[jira] [Updated] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23132: - Description: Seems currently we don't actually run the doctests in  {{ml.image.py}}. It'd be bette

[jira] [Commented] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328714#comment-16328714 ] Apache Spark commented on SPARK-23132: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23132: Assignee: Apache Spark > Run ml.image doctests in tests > -- >

[jira] [Assigned] (SPARK-23132) Run ml.image doctests in tests

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23132: Assignee: (was: Apache Spark) > Run ml.image doctests in tests > -

[jira] [Commented] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328734#comment-16328734 ] Marco Gaido commented on SPARK-23130: - The "_resources" files leak should have been f

[jira] [Resolved] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-23130. - Resolution: Duplicate > Spark Thrift does not clean-up temporary files (/tmp/*_resources and > /

[jira] [Commented] (SPARK-15401) Spark Thrift server creates empty directories in tmp directory on the driver

2018-01-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328738#comment-16328738 ] Marco Gaido commented on SPARK-15401: - this should have been fixed in SPARK-22793. >

[jira] [Resolved] (SPARK-15401) Spark Thrift server creates empty directories in tmp directory on the driver

2018-01-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-15401. - Resolution: Duplicate > Spark Thrift server creates empty directories in tmp directory on the dri

[jira] [Resolved] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23125. --- Resolution: Not A Problem Probably, but this is a Kafka config issue. If you're not using matched Kaf

[jira] [Resolved] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23123. --- Resolution: Not A Problem > Unable to run Spark Job with Hadoop NameNode Federation using ViewFS > --

[jira] [Resolved] (SPARK-23126) I used the Project operator and modified the source. After compiling successfully, and testing the jars, I got the exception. Maybe the phenomenon is related with impli

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23126. --- Resolution: Invalid Fix Version/s: (was: 2.2.0) Target Version/s: (was: 2.2.0)

[jira] [Commented] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UTF from HDFS

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328777#comment-16328777 ] Sean Owen commented on SPARK-21697: --- Isn't this an HDFS problem? what could Spark do ab

[jira] [Commented] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread zhaoshijie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328782#comment-16328782 ] zhaoshijie commented on SPARK-23125: spark 2.2 use kafka version is 0.10.0.1 and I do

[jira] [Commented] (SPARK-23125) Offset commit failed when spark-streaming batch time is more than kafkaParams session timeout.

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328787#comment-16328787 ] Sean Owen commented on SPARK-23125: --- The error message you cite, which is from the vers

[jira] [Assigned] (SPARK-21783) Turn on ORC filter push-down by default

2018-01-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21783: --- Assignee: Dongjoon Hyun > Turn on ORC filter push-down by default >

[jira] [Resolved] (SPARK-21783) Turn on ORC filter push-down by default

2018-01-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21783. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20265 [https://githu

[jira] [Resolved] (SPARK-23076) When we call cache() on RDD which depends on ShuffleRowRDD, we will get an error result

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23076. --- Resolution: Not A Problem > When we call cache() on RDD which depends on ShuffleRowRDD, we will get a

[jira] [Commented] (SPARK-23076) When we call cache() on RDD which depends on ShuffleRowRDD, we will get an error result

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328801#comment-16328801 ] Sean Owen commented on SPARK-23076: --- You're relying on behavior that this class doesn't

[jira] [Commented] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328868#comment-16328868 ] Sean Owen commented on SPARK-23131: --- This requires updating Twitter Chill too, really,

[jira] [Commented] (SPARK-22886) ML test for StructuredStreaming: spark.ml.recommendation

2018-01-17 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328874#comment-16328874 ] Gabor Somogyi commented on SPARK-22886: --- I would like to work on this. Please notif

[jira] [Commented] (SPARK-22884) ML test for StructuredStreaming: spark.ml.clustering

2018-01-17 Thread Sandor Murakozi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328898#comment-16328898 ] Sandor Murakozi commented on SPARK-22884: - Is there anybody working on this? If

[jira] [Commented] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UTF from HDFS

2018-01-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328928#comment-16328928 ] Steve Loughran commented on SPARK-21697: No, it's spark's ability to have hdfs://

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

2018-01-17 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328936#comment-16328936 ] Li Jin commented on SPARK-22980: I agree with [~cloud_fan]. I think it's enough to docume

[jira] [Commented] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329065#comment-16329065 ] Marcelo Vanzin commented on SPARK-23020: Bummer. I'll try to take another look la

[jira] [Assigned] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23020: Assignee: Marcelo Vanzin (was: Apache Spark) > Re-enable Flaky Test: > org.apache.spark.

[jira] [Assigned] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23020: Assignee: Apache Spark (was: Marcelo Vanzin) > Re-enable Flaky Test: > org.apache.spark.

[jira] [Created] (SPARK-23133) Spark options are not passed to the Executor in Docker context

2018-01-17 Thread Andrew Korzhuev (JIRA)
Andrew Korzhuev created SPARK-23133: --- Summary: Spark options are not passed to the Executor in Docker context Key: SPARK-23133 URL: https://issues.apache.org/jira/browse/SPARK-23133 Project: Spark

[jira] [Commented] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329100#comment-16329100 ] Peigen commented on SPARK-23131: I realize this happens when I try to serialize the model

[jira] [Updated] (SPARK-23131) Stackoverflow using ML and Kryo serializer

2018-01-17 Thread Peigen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peigen updated SPARK-23131: --- Description: When trying to use GeneralizedLinearRegression model and set SparkConf to use KryoSerializer(Ja

[jira] [Created] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2018-01-17 Thread Shahid K I (JIRA)
Shahid K I created SPARK-23134: -- Summary: WebUI is showing the cache table details even after cache idle timeout Key: SPARK-23134 URL: https://issues.apache.org/jira/browse/SPARK-23134 Project: Spark

[jira] [Updated] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2018-01-17 Thread Shahid K I (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shahid K I updated SPARK-23134: --- Environment:  Run Cache command with below configuration to cache the RDD blocks  spark.dynami

[jira] [Updated] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2018-01-17 Thread Shahid K I (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shahid K I updated SPARK-23134: --- Environment:  Run Cache command with below configuration to cache the RDD blocks  spark.dynamicAllo

[jira] [Updated] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2018-01-17 Thread Shahid K I (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shahid K I updated SPARK-23134: --- Description: After cachedExecutorIdleTimeout, WebUI shows the cached partition details in the storage

[jira] [Commented] (SPARK-8682) Range Join for Spark SQL

2018-01-17 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329150#comment-16329150 ] Ruslan Dautkhanov commented on SPARK-8682: -- Range joins need some serious optimiz

[jira] [Commented] (SPARK-23011) Support alternative function form with group aggregate pandas UDF

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329177#comment-16329177 ] Apache Spark commented on SPARK-23011: -- User 'icexelloss' has created a pull request

[jira] [Commented] (SPARK-23133) Spark options are not passed to the Executor in Docker context

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329189#comment-16329189 ] Apache Spark commented on SPARK-23133: -- User 'andrusha' has created a pull request f

[jira] [Assigned] (SPARK-23133) Spark options are not passed to the Executor in Docker context

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23133: Assignee: Apache Spark > Spark options are not passed to the Executor in Docker context >

[jira] [Assigned] (SPARK-23133) Spark options are not passed to the Executor in Docker context

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23133: Assignee: (was: Apache Spark) > Spark options are not passed to the Executor in Docker

[jira] [Created] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23135: --- Summary: Accumulators don't show up properly in the Stages page anymore Key: SPARK-23135 URL: https://issues.apache.org/jira/browse/SPARK-23135 Project: Spark

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Attachment: webUIAccumulatorRegression.png > Accumulators don't show up properly in the Stages page

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Description: Didn't do a lot of digging but may be caused by: [https://github.com/apache/spark/com

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Environment:       was: Didn't do a lot of digging but may be caused by: [https://github.com/

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329293#comment-16329293 ] Burak Yavuz commented on SPARK-23135: - cc [~vanzin] > Accumulators don't show up pro

[jira] [Commented] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329306#comment-16329306 ] Apache Spark commented on SPARK-23020: -- User 'vanzin' has created a pull request for

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329353#comment-16329353 ] Marcelo Vanzin commented on SPARK-23135: I'll try to take a look at the code, but

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329362#comment-16329362 ] Marcelo Vanzin commented on SPARK-23135: (By fine I mean the table renders correc

[jira] [Commented] (SPARK-23103) LevelDB store not iterating correctly when indexed value has negative value

2018-01-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329367#comment-16329367 ] Imran Rashid commented on SPARK-23103: -- [~vanzin] is this really minor, not a blocke

[jira] [Comment Edited] (SPARK-23103) LevelDB store not iterating correctly when indexed value has negative value

2018-01-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329367#comment-16329367 ] Imran Rashid edited comment on SPARK-23103 at 1/17/18 8:07 PM:

[jira] [Commented] (SPARK-23103) LevelDB store not iterating correctly when indexed value has negative value

2018-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329380#comment-16329380 ] Marcelo Vanzin commented on SPARK-23103: Given the unit test failure in the PR it

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329398#comment-16329398 ] Marcelo Vanzin commented on SPARK-23135: Nevermind, I was able to get the wrong t

[jira] [Commented] (SPARK-22976) Worker cleanup can remove running driver directories

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329400#comment-16329400 ] Apache Spark commented on SPARK-22976: -- User 'RussellSpitzer' has created a pull req

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329419#comment-16329419 ] Apache Spark commented on SPARK-23135: -- User 'vanzin' has created a pull request for

[jira] [Assigned] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23135: Assignee: Apache Spark > Accumulators don't show up properly in the Stages page anymore >

[jira] [Assigned] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23135: Assignee: (was: Apache Spark) > Accumulators don't show up properly in the Stages page

[jira] [Commented] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329464#comment-16329464 ] Sean Roberts commented on SPARK-23130: -- Marco - Which JIRA resolves the pipeout issu

[jira] [Reopened] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

2018-01-17 Thread Sean Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Roberts reopened SPARK-23130: -- > Spark Thrift does not clean-up temporary files (/tmp/*_resources and > /tmp/hive/*.pipeout) > --

[jira] [Commented] (SPARK-23114) Spark R 2.3 QA umbrella

2018-01-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329478#comment-16329478 ] Joseph K. Bradley commented on SPARK-23114: --- Thank you! > Spark R 2.3 QA umbre

[jira] [Assigned] (SPARK-23114) Spark R 2.3 QA umbrella

2018-01-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-23114: - Assignee: Felix Cheung > Spark R 2.3 QA umbrella > --- > >

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329481#comment-16329481 ] Bryan Cutler commented on SPARK-23109: -- [~josephkb] I can take this, thanks! > ML 2

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329489#comment-16329489 ] Joseph K. Bradley commented on SPARK-23110: --- [~WeichenXu123] said he'd take thi

  1   2   >