[jira] [Created] (SPARK-27802) SparkUI throws NoSuchElementException when inconsistency appears between `ExecutorStageSummaryWrapper`s and `ExecutorSummaryWrapper`s

2019-05-21 Thread liupengcheng (JIRA)
liupengcheng created SPARK-27802: Summary: SparkUI throws NoSuchElementException when inconsistency appears between `ExecutorStageSummaryWrapper`s and `ExecutorSummaryWrapper`s Key: SPARK-27802 URL: https://issues

[jira] [Updated] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

2019-05-21 Thread Steven Rand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated SPARK-27773: Description: The health of the external shuffle service is currently difficult to monitor. At lea

[jira] [Commented] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-05-21 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845559#comment-16845559 ] Ohad Raviv commented on SPARK-18748: [~kelemen] - thanks for sharing. > UDF multipl

[jira] [Closed] (SPARK-16820) Sparse - Sparse matrix multiplication

2019-05-21 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ohad Raviv closed SPARK-16820. -- resolved. > Sparse - Sparse matrix multiplication > - > >

[jira] [Commented] (SPARK-16820) Sparse - Sparse matrix multiplication

2019-05-21 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845557#comment-16845557 ] Ohad Raviv commented on SPARK-16820:  this issue was resolved by SPARK-19368 and SPA

[jira] [Assigned] (SPARK-27801) InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27801: Assignee: Apache Spark > InMemoryFileIndex.listLeafFiles should use listLocatedStatus for

[jira] [Assigned] (SPARK-27801) InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27801: Assignee: (was: Apache Spark) > InMemoryFileIndex.listLeafFiles should use listLocate

[jira] [Created] (SPARK-27801) InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem

2019-05-21 Thread Rob Russo (JIRA)
Rob Russo created SPARK-27801: - Summary: InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem Key: SPARK-27801 URL: https://issues.apache.org/jira/browse/SPARK-27801 Proj

[jira] [Commented] (SPARK-27797) Shuffle service metric "registeredConnections" not tracked correctly

2019-05-21 Thread Steven Rand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845535#comment-16845535 ] Steven Rand commented on SPARK-27797: - I think it might be okay -- ExternalShuffleSe

[jira] [Resolved] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27800. --- Resolution: Fixed Assignee: Alex Liu Fix Version/s: 3.0.0

[jira] [Updated] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27800: -- Component/s: SQL > Example for xor function has a wrong answer > -

[jira] [Resolved] (SPARK-27778) toPandas with arrow enabled fails for DF with no partitions

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27778. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24650 [https://gi

[jira] [Assigned] (SPARK-27778) toPandas with arrow enabled fails for DF with no partitions

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27778: Assignee: David Vogelbacher > toPandas with arrow enabled fails for DF with no partitions

[jira] [Resolved] (SPARK-24586) Upcast should not allow casting from string to other types

2019-05-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24586. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21586 [https://gith

[jira] [Assigned] (SPARK-27698) Add new method for getting pushed down filters in Parquet file reader

2019-05-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27698: --- Assignee: Gengliang Wang > Add new method for getting pushed down filters in Parquet file r

[jira] [Resolved] (SPARK-27698) Add new method for getting pushed down filters in Parquet file reader

2019-05-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27698. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24597 [https://gith

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845456#comment-16845456 ] Hyukjin Kwon commented on SPARK-27798: -- Seems fixed in the current master: ``` +--

[jira] [Commented] (SPARK-18406) Race between end-of-task and completion iterator read lock release

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845431#comment-16845431 ] Apache Spark commented on SPARK-18406: -- User 'rezasafi' has created a pull request

[jira] [Issue Comment Deleted] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-27800: - Comment: was deleted (was:  A PR created to fix this, Would anyone take a look. :) [https://github.com/

[jira] [Commented] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845429#comment-16845429 ] Alex Liu commented on SPARK-27800: --  A PR created to fix this, Would anyone take a look

[jira] [Updated] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-27800: - Description: See [https://spark.apache.org/docs/latest/api/sql/index.html#_14] 3 ^ 5 should be 6 rather

[jira] [Updated] (SPARK-27792) SkewJoin--handle only skewed keys with broadcastjoin and other keys with normal join

2019-05-21 Thread Jason Guo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Guo updated SPARK-27792: -- Description: This feature is designed to handle data skew in Join   *Senario* * A big table (big_sk

[jira] [Assigned] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27800: Assignee: (was: Apache Spark) > Example for xor function has a wrong answer > ---

[jira] [Assigned] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27800: Assignee: Apache Spark > Example for xor function has a wrong answer > --

[jira] [Updated] (SPARK-27792) SkewJoin--handle only skewed keys with broadcastjoin and other keys with normal join

2019-05-21 Thread Jason Guo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Guo updated SPARK-27792: -- Shepherd: Liang-Chi Hsieh > SkewJoin--handle only skewed keys with broadcastjoin and other keys with

[jira] [Created] (SPARK-27800) Example for xor function has a wrong answer

2019-05-21 Thread Alex Liu (JIRA)
Alex Liu created SPARK-27800: Summary: Example for xor function has a wrong answer Key: SPARK-27800 URL: https://issues.apache.org/jira/browse/SPARK-27800 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-25139) PythonRunner#WriterThread released block after TaskRunner finally block which invoke BlockManager#releaseAllLocksForTask

2019-05-21 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845422#comment-16845422 ] Reza Safi commented on SPARK-25139: --- Sure, I will send a pr soon. Thanks. > PythonRun

[jira] [Resolved] (SPARK-27774) Avoid hardcoded configs

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27774. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24631 [https://gi

[jira] [Assigned] (SPARK-27774) Avoid hardcoded configs

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27774: Assignee: wenxuanguan > Avoid hardcoded configs > --- > >

[jira] [Resolved] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27737. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24620 [https://gi

[jira] [Assigned] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27737: Assignee: Yuming Wang (was: yuming.wang) > Upgrade to 2.3.5 for Hive Metastore Client 2.

[jira] [Assigned] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27737: Assignee: yuming.wang > Upgrade to 2.3.5 for Hive Metastore Client 2.3 >

[jira] [Commented] (SPARK-25139) PythonRunner#WriterThread released block after TaskRunner finally block which invoke BlockManager#releaseAllLocksForTask

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845401#comment-16845401 ] Hyukjin Kwon commented on SPARK-25139: -- I think we can. Please go ahead and open a

[jira] [Commented] (SPARK-27599) DataFrameWriter.partitionBy should be optional when writing to a hive table

2019-05-21 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845378#comment-16845378 ] Nick Dimiduk commented on SPARK-27599: -- Sure [~Alexander_Fedosov]. Hive DDL let's y

[jira] [Updated] (SPARK-27799) Allow SerializerManager.canUseKryo whitelist to be extended via a configuration

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27799: --- Issue Type: New Feature (was: Bug) > Allow SerializerManager.canUseKryo whitelist to be extended vi

[jira] [Updated] (SPARK-27799) Allow SerializerManager.canUseKryo whitelist to be extended via a configuration

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27799: --- Description: Kryo serialization can offer a substantial performance boost compared to Java serializ

[jira] [Updated] (SPARK-27799) Allow SerializerManager.canUseKryo whitelist to be extended via a configuration

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27799: --- Description: Kryo serialization can offer a substantial performance boost compared to Java serializ

[jira] [Updated] (SPARK-27799) Allow SerializerManager.canUseKryo whitelist to be extended via a configuration

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27799: --- Description: Kryo serialization can offer a substantial performance boost compared to Java serializ

[jira] [Created] (SPARK-27799) Allow SerializerManager.canUseKryo to be customized via configuration

2019-05-21 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27799: -- Summary: Allow SerializerManager.canUseKryo to be customized via configuration Key: SPARK-27799 URL: https://issues.apache.org/jira/browse/SPARK-27799 Project: Spark

[jira] [Updated] (SPARK-27799) Allow SerializerManager.canUseKryo whitelist to be extended via a configuration

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27799: --- Summary: Allow SerializerManager.canUseKryo whitelist to be extended via a configuration (was: Allo

[jira] [Commented] (SPARK-16738) Queryable state for Spark State Store

2019-05-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845364#comment-16845364 ] Hyukjin Kwon commented on SPARK-16738: -- No feedback might imply that users actually

[jira] [Commented] (SPARK-23978) Kryo much slower when mllib jar not on classpath

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845336#comment-16845336 ] Josh Rosen commented on SPARK-23978: +1; I've also seen this in unit tests of my own

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Description: Steps to reproduce: Create a local Dataset (at least two distinct rows) with a binar

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845311#comment-16845311 ] Yosuke Mori commented on SPARK-27798: - Someone else seems to have encountered this i

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Description: Steps to reproduce: Create a local Dataset (at least two distinct rows) with a binar

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Summary: from_avro can modify variables in other rows in local mode (was: from_avro can modify va

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Labels: correctness (was: ) > from_avro can modify variables in other rows >

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Description: Steps to reproduce: Create a local Dataset (at least two distinct rows) with a binar

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Description: Steps to reproduce: Create a local Dataset (at least two distinct rows) with a binar

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Description: Steps to reproduce: Create a local Dataset (at least two distinct rows) with a binar

[jira] [Updated] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Mori updated SPARK-27798: Attachment: Screen Shot 2019-05-21 at 2.39.27 PM.png > from_avro can modify variables in other row

[jira] [Created] (SPARK-27798) from_avro can modify variables in other rows

2019-05-21 Thread Yosuke Mori (JIRA)
Yosuke Mori created SPARK-27798: --- Summary: from_avro can modify variables in other rows Key: SPARK-27798 URL: https://issues.apache.org/jira/browse/SPARK-27798 Project: Spark Issue Type: Bug

[jira] [Reopened] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-20547: -- Assignee: Shixiong Zhu > ExecutorClassLoader's findClass may not work correctly when a task

[jira] [Comment Edited] (SPARK-27768) Infinity, -Infinity, NaN should be recognized in a case insensitive manner

2019-05-21 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845277#comment-16845277 ] Dilip Biswal edited comment on SPARK-27768 at 5/21/19 9:48 PM: ---

[jira] [Commented] (SPARK-27768) Infinity, -Infinity, NaN should be recognized in a case insensitive manner

2019-05-21 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845277#comment-16845277 ] Dilip Biswal commented on SPARK-27768: -- [~smilegator] I will wait for the test PR t

[jira] [Commented] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845249#comment-16845249 ] Josh Rosen commented on SPARK-27676: +1 [~ste...@apache.org]: I agree that this is b

[jira] [Assigned] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27676: Assignee: Apache Spark > InMemoryFileIndex should hard-fail on missing files instead of l

[jira] [Assigned] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27676: Assignee: (was: Apache Spark) > InMemoryFileIndex should hard-fail on missing files i

[jira] [Assigned] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-27676: -- Assignee: Josh Rosen > InMemoryFileIndex should hard-fail on missing files instead of logging

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-05-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845219#comment-16845219 ] Thomas Graves commented on SPARK-27495: --- I'm working though the design of this and

[jira] [Created] (SPARK-27797) Shuffle service metric "registeredConnections" not tracked correctly

2019-05-21 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-27797: -- Summary: Shuffle service metric "registeredConnections" not tracked correctly Key: SPARK-27797 URL: https://issues.apache.org/jira/browse/SPARK-27797 Project: Spa

[jira] [Commented] (SPARK-16738) Queryable state for Spark State Store

2019-05-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845170#comment-16845170 ] Stavros Kontopoulos commented on SPARK-16738: - @[~hyukjin.kwon] I think this

[jira] [Commented] (SPARK-25139) PythonRunner#WriterThread released block after TaskRunner finally block which invoke BlockManager#releaseAllLocksForTask

2019-05-21 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845136#comment-16845136 ] Reza Safi commented on SPARK-25139: --- Any reason that this bug fix wasn't merged to 2.3

[jira] [Resolved] (SPARK-27248) REFRESH TABLE should recreate cache with same cache name and storage level

2019-05-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27248. --- Resolution: Fixed Assignee: William Wong Fix Version/s: 3.0.0 This is resolv

[jira] [Updated] (SPARK-27439) Explainging Dataset should show correct resolved plans

2019-05-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-27439: Issue Type: Bug (was: Improvement) > Explainging Dataset should show correct resolved plans > ---

[jira] [Resolved] (SPARK-27439) Explainging Dataset should show correct resolved plans

2019-05-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27439. - Resolution: Fixed Fix Version/s: 3.0.0 > Explainging Dataset should show correct resolved plans >

[jira] [Resolved] (SPARK-27796) Remove obsolete spark-mesos Docker image

2019-05-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27796. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via https://github.com/apache

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845109#comment-16845109 ] David C Navas commented on SPARK-27726: --- sub-tasks closed as dupes against this um

[jira] [Resolved] (SPARK-27731) Cleanup some non-compile time type checking and exception handling

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas resolved SPARK-27731. --- Resolution: Duplicate Fix Version/s: 3.0.0 resolved in umbrella ticket > Cleanup som

[jira] [Resolved] (SPARK-27729) Extract deletion of the summaries from the stage deletion loop

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas resolved SPARK-27729. --- Resolution: Duplicate Fix Version/s: 3.0.0 resolved in umbrella ticket > Extract del

[jira] [Resolved] (SPARK-27730) Add support for removeAllKeys

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas resolved SPARK-27730. --- Resolution: Duplicate Fix Version/s: 3.0.0 resolved in umbrella ticket > Add support

[jira] [Resolved] (SPARK-27728) Address thread-safety of InMemoryStore and ElementTrackingStores.

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas resolved SPARK-27728. --- Resolution: Duplicate Fix Version/s: 3.0.0 Resolved in umbrella ticket > Address thr

[jira] [Resolved] (SPARK-27727) Asynchronous ElementStore cleanup should have only one pending cleanup per class

2019-05-21 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas resolved SPARK-27727. --- Resolution: Duplicate Fix Version/s: 3.0.0 > Asynchronous ElementStore cleanup should

[jira] [Updated] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-27726: --- Fix Version/s: 2.4.4 > Performance of InMemoryStore suffers under load > ---

[jira] [Reopened] (SPARK-11095) Simplify Netty RPC implementation by using a separate thread pool for each endpoint

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-11095: -- > Simplify Netty RPC implementation by using a separate thread pool for each > endpoint > ---

[jira] [Resolved] (SPARK-11095) Simplify Netty RPC implementation by using a separate thread pool for each endpoint

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-11095. -- Resolution: Won't Do > Simplify Netty RPC implementation by using a separate thread pool for e

[jira] [Reopened] (SPARK-17858) Provide option for Spark SQL to skip corrupt files

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-17858: -- Assignee: Shixiong Zhu > Provide option for Spark SQL to skip corrupt files > --

[jira] [Resolved] (SPARK-17858) Provide option for Spark SQL to skip corrupt files

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17858. -- Resolution: Duplicate > Provide option for Spark SQL to skip corrupt files > -

[jira] [Updated] (SPARK-10719) SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala 2.10

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-10719: - Fix Version/s: 2.3.0 > SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala 2.1

[jira] [Closed] (SPARK-10719) SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala 2.10

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu closed SPARK-10719. Assignee: Shixiong Zhu We can close this since Scala 2.10 has been dropped in Spark 2.3.0. > SQLI

[jira] [Resolved] (SPARK-27762) Support user provided avro schema for writing fields with different ordering

2019-05-21 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-27762. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24635 [https://github.com/a

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845061#comment-16845061 ] Marcelo Vanzin commented on SPARK-27726: [~davidnavas] all of the sub-tasks were

[jira] [Assigned] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27726: -- Assignee: David C Navas > Performance of InMemoryStore suffers under load > -

[jira] [Resolved] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27726. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24616 [https:

[jira] [Assigned] (SPARK-23626) Spark DAGScheduler scheduling performance hindered on JobSubmitted Event

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23626: Assignee: (was: Apache Spark) > Spark DAGScheduler scheduling performance hindered on

[jira] [Assigned] (SPARK-23626) Spark DAGScheduler scheduling performance hindered on JobSubmitted Event

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23626: Assignee: Apache Spark > Spark DAGScheduler scheduling performance hindered on JobSubmitt

[jira] [Assigned] (SPARK-27796) Remove obsolete spark-mesos Docker image

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27796: Assignee: Sean Owen (was: Apache Spark) > Remove obsolete spark-mesos Docker image > ---

[jira] [Assigned] (SPARK-27796) Remove obsolete spark-mesos Docker image

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27796: Assignee: Apache Spark (was: Sean Owen) > Remove obsolete spark-mesos Docker image > ---

[jira] [Created] (SPARK-27796) Remove obsolete spark-mesos Docker image

2019-05-21 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27796: - Summary: Remove obsolete spark-mesos Docker image Key: SPARK-27796 URL: https://issues.apache.org/jira/browse/SPARK-27796 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-27790) Support SQL INTERVAL types

2019-05-21 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-27790: --- Description: SQL standard defines 2 interval types: # year-month interval contains a YEAR field or a

[jira] [Updated] (SPARK-27790) Support SQL INTERVAL types

2019-05-21 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-27790: --- Description: SQL standard defines 2 interval types: # year-month interval contains a YEAR field or a

[jira] [Closed] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2019-05-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21349. - > Make TASK_SIZE_TO_WARN_KB configurable > -- > >

[jira] [Updated] (SPARK-27779) Regression when explode on map in Generate

2019-05-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27779: Description: When I ran MiscBenchmark for SPARK-27707, I found a regression regarding exp

[jira] [Updated] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Bouriakov updated SPARK-27795: Environment: (was: Steps to reproduce: In a Servlet, instantiate a local spark

[jira] [Updated] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Bouriakov updated SPARK-27795: Description: Static files of the Web UI from *spark-core_2.12-2.4.3.jar/org/apach

[jira] [Updated] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Bouriakov updated SPARK-27795: Attachment: broken_ui.PNG > Spark Web UI is broken when running in local mode with

[jira] [Commented] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844893#comment-16844893 ] Alexander Bouriakov commented on SPARK-27795: - My ugly workaround currently

[jira] [Updated] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Bouriakov updated SPARK-27795: Description: Static files of the Web UI from *spark-core_2.12-2.4.3.jar/org/apach

[jira] [Created] (SPARK-27795) Spark Web UI is broken when running in local mode within WildFly application server

2019-05-21 Thread Alexander Bouriakov (JIRA)
Alexander Bouriakov created SPARK-27795: --- Summary: Spark Web UI is broken when running in local mode within WildFly application server Key: SPARK-27795 URL: https://issues.apache.org/jira/browse/SPARK-27795

[jira] [Assigned] (SPARK-27794) Use secure URLs for downloading CRAN artifacts

2019-05-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27794: Assignee: Apache Spark (was: Sean Owen) > Use secure URLs for downloading CRAN artifacts

  1   2   >