[jira] [Commented] (SPARK-20866) Dataset map does not respect nullable field

2017-05-25 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024350#comment-16024350 ] Takeshi Yamamuro commented on SPARK-20866: -- SPARK-18284 [~kiszk] fixed also reso

[jira] [Closed] (SPARK-20866) Dataset map does not respect nullable field

2017-05-25 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-20866. Resolution: Fixed Fix Version/s: 2.2.0 > Dataset map does not respect nullable field

[jira] [Updated] (SPARK-20876) If the input parameter is float type for ceil or floor ,the result is not we expected

2017-05-25 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-20876: Summary: If the input parameter is float type for ceil or floor ,the result is not we expected (was: if t

[jira] [Resolved] (SPARK-20879) Spark SQL will read Date type column from avro file as Int

2017-05-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20879. -- Resolution: Invalid I think this is an issue in https://github.com/databricks/spark-avro not wi

[jira] [Resolved] (SPARK-20250) Improper OOM error when a task been killed while spilling data

2017-05-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20250. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.2 2.0.3 >

[jira] [Assigned] (SPARK-20250) Improper OOM error when a task been killed while spilling data

2017-05-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20250: --- Assignee: coneyliu > Improper OOM error when a task been killed while spilling data > --

[jira] [Resolved] (SPARK-20878) Pyspark date string parsing erroneously treats 1 as 10

2017-05-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20878. -- Resolution: Not A Problem I guess it compares a (casted by type coercion rule) string to a stri

[jira] [Assigned] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-05-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19659: --- Assignee: jin xing > Fetch big blocks to disk when shuffle-read > --

[jira] [Resolved] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-05-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19659. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16989 [https://githu

[jira] [Created] (SPARK-20880) When spark SQL is used with Avro-backed HIVE tables, NPE from org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories.

2017-05-25 Thread Vinod KC (JIRA)
Vinod KC created SPARK-20880: Summary: When spark SQL is used with Avro-backed HIVE tables, NPE from org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories. Key: SPARK-20880 URL: https:

[jira] [Updated] (SPARK-20880) When spark SQL is used with Avro-backed HIVE tables, NPE from org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories.

2017-05-25 Thread Vinod KC (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-20880: - Description: When spark SQL is used with Avro-backed HIVE tables, intermittently getting NPE from org

[jira] [Updated] (SPARK-14174) Accelerate KMeans via Mini-Batch EM

2017-05-25 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-14174: - Attachment: MiniBatchKMeans_Performance_II.pdf > Accelerate KMeans via Mini-Batch EM > --

[jira] [Commented] (SPARK-14174) Accelerate KMeans via Mini-Batch EM

2017-05-25 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024417#comment-16024417 ] zhengruifeng commented on SPARK-14174: -- [~mlnick] OK, I have updated the attached pe

[jira] [Commented] (SPARK-14174) Accelerate KMeans via Mini-Batch EM

2017-05-25 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024428#comment-16024428 ] Nick Pentreath commented on SPARK-14174: Yes, as one would expect. But the key re

[jira] [Resolved] (SPARK-20834) TypeCoercion:loss of precision when widening long and float,or int and float

2017-05-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20834. --- Resolution: Not A Problem > TypeCoercion:loss of precision when widening long and float,or int and fl

[jira] [Created] (SPARK-20881) Use Hive's stats in metastore when cbo is disabled

2017-05-25 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-20881: Summary: Use Hive's stats in metastore when cbo is disabled Key: SPARK-20881 URL: https://issues.apache.org/jira/browse/SPARK-20881 Project: Spark Issue Type

[jira] [Assigned] (SPARK-20881) Use Hive's stats in metastore when cbo is disabled

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20881: Assignee: Apache Spark > Use Hive's stats in metastore when cbo is disabled >

[jira] [Commented] (SPARK-20881) Use Hive's stats in metastore when cbo is disabled

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024456#comment-16024456 ] Apache Spark commented on SPARK-20881: -- User 'wzhfy' has created a pull request for

[jira] [Assigned] (SPARK-20881) Use Hive's stats in metastore when cbo is disabled

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20881: Assignee: (was: Apache Spark) > Use Hive's stats in metastore when cbo is disabled > -

[jira] [Comment Edited] (SPARK-20878) Pyspark date string parsing erroneously treats 1 as 10

2017-05-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024378#comment-16024378 ] Hyukjin Kwon edited comment on SPARK-20878 at 5/25/17 9:51 AM:

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Attachment: executor_jstack > Executor is waiting for ShuffleBlockFetcherIterator > ---

[jira] [Created] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
cen yuhai created SPARK-20882: - Summary: Executor is waiting for ShuffleBlockFetcherIterator Key: SPARK-20882 URL: https://issues.apache.org/jira/browse/SPARK-20882 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Attachment: screenshot-1.png > Executor is waiting for ShuffleBlockFetcherIterator > --

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Attachment: screenshot-2.png > Executor is waiting for ShuffleBlockFetcherIterator > --

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Attachment: executor_log > Executor is waiting for ShuffleBlockFetcherIterator > --

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Attachment: screenshot-3.png > Executor is waiting for ShuffleBlockFetcherIterator > --

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024503#comment-16024503 ] cen yuhai commented on SPARK-20882: --- [~zsxwing] > Executor is waiting for ShuffleBlock

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Description: This bug is like https://issues.apache.org/jira/browse/SPARK-19300. but I have updated my

[jira] [Commented] (SPARK-8578) Should ignore user defined output committer when appending data

2017-05-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024526#comment-16024526 ] Steve Loughran commented on SPARK-8578: --- Given SPARK-10063 has pulled the {{DirectPa

[jira] [Assigned] (SPARK-20376) Make StateStoreProvider plugable

2017-05-25 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-20376: - Assignee: Tathagata Das > Make StateStoreProvider plugable > ---

[jira] [Updated] (SPARK-20376) Make StateStoreProvider plugable

2017-05-25 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-20376: -- Target Version/s: 2.3.0 > Make StateStoreProvider plugable > >

[jira] [Commented] (SPARK-20754) Add Function Alias For MOD/TRUNCT/POSITION

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024543#comment-16024543 ] Apache Spark commented on SPARK-20754: -- User 'wangyum' has created a pull request fo

[jira] [Assigned] (SPARK-20754) Add Function Alias For MOD/TRUNCT/POSITION

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20754: Assignee: Apache Spark > Add Function Alias For MOD/TRUNCT/POSITION >

[jira] [Assigned] (SPARK-20754) Add Function Alias For MOD/TRUNCT/POSITION

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20754: Assignee: (was: Apache Spark) > Add Function Alias For MOD/TRUNCT/POSITION > -

[jira] [Created] (SPARK-20883) Improve StateStore APIs for efficiency

2017-05-25 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-20883: - Summary: Improve StateStore APIs for efficiency Key: SPARK-20883 URL: https://issues.apache.org/jira/browse/SPARK-20883 Project: Spark Issue Type: Improvem

[jira] [Updated] (SPARK-20691) Difference between Storage Memory as seen internally and in web UI

2017-05-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20691: -- Labels: starter (was: ) > Difference between Storage Memory as seen internally and in web UI > ---

[jira] [Commented] (SPARK-20883) Improve StateStore APIs for efficiency

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024563#comment-16024563 ] Apache Spark commented on SPARK-20883: -- User 'tdas' has created a pull request for t

[jira] [Assigned] (SPARK-20376) Make StateStoreProvider plugable

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20376: Assignee: Apache Spark (was: Tathagata Das) > Make StateStoreProvider plugable >

[jira] [Assigned] (SPARK-20883) Improve StateStore APIs for efficiency

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20883: Assignee: Tathagata Das (was: Apache Spark) > Improve StateStore APIs for efficiency > --

[jira] [Commented] (SPARK-20376) Make StateStoreProvider plugable

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024564#comment-16024564 ] Apache Spark commented on SPARK-20376: -- User 'tdas' has created a pull request for t

[jira] [Assigned] (SPARK-20883) Improve StateStore APIs for efficiency

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20883: Assignee: Apache Spark (was: Tathagata Das) > Improve StateStore APIs for efficiency > --

[jira] [Assigned] (SPARK-20376) Make StateStoreProvider plugable

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20376: Assignee: Tathagata Das (was: Apache Spark) > Make StateStoreProvider plugable >

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

2017-05-25 Thread Yuval Itzchakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024602#comment-16024602 ] Yuval Itzchakov commented on SPARK-19067: - [~marmbrus] [~tdas] Is there any reas

[jira] [Created] (SPARK-20884) Spark' masters will be both standby due to the bug of curator

2017-05-25 Thread liuzhaokun (JIRA)
liuzhaokun created SPARK-20884: -- Summary: Spark' masters will be both standby due to the bug of curator Key: SPARK-20884 URL: https://issues.apache.org/jira/browse/SPARK-20884 Project: Spark I

[jira] [Commented] (SPARK-20884) Spark' masters will be both standby due to the bug of curator

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024623#comment-16024623 ] Apache Spark commented on SPARK-20884: -- User 'liu-zhaokun' has created a pull reques

[jira] [Assigned] (SPARK-20884) Spark' masters will be both standby due to the bug of curator

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20884: Assignee: (was: Apache Spark) > Spark' masters will be both standby due to the bug of

[jira] [Assigned] (SPARK-20884) Spark' masters will be both standby due to the bug of curator

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20884: Assignee: Apache Spark > Spark' masters will be both standby due to the bug of curator >

[jira] [Updated] (SPARK-20881) Use Hive's stats in metastore when cbo is disabled

2017-05-25 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20881: - Description: Currently statistics are generated by "analyze command" in Spark. However, when us

[jira] [Updated] (SPARK-18838) High latency of event processing for large jobs

2017-05-25 Thread Antoine PRANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine PRANG updated SPARK-18838: -- Attachment: perfResults.pdf last timings > High latency of event processing for large jobs >

[jira] [Updated] (SPARK-19293) Spark 2.1.x unstable with spark.speculation=true

2017-05-25 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Momot updated SPARK-19293: - Affects Version/s: 2.1.1 Component/s: Spark Core Summary: Spark 2.1.x unst

[jira] [Commented] (SPARK-19293) Spark 2.1.x unstable with spark.speculation=true

2017-05-25 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024684#comment-16024684 ] Damian Momot commented on SPARK-19293: -- >From what I can see speculative tasks are f

[jira] [Comment Edited] (SPARK-19293) Spark 2.1.x unstable with spark.speculation=true

2017-05-25 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024684#comment-16024684 ] Damian Momot edited comment on SPARK-19293 at 5/25/17 12:44 PM: ---

[jira] [Comment Edited] (SPARK-19293) Spark 2.1.x unstable with spark.speculation=true

2017-05-25 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024684#comment-16024684 ] Damian Momot edited comment on SPARK-19293 at 5/25/17 12:45 PM: ---

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-05-25 Thread Antoine PRANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024690#comment-16024690 ] Antoine PRANG commented on SPARK-18838: --- [~joshrosen] I uploaded the new timings an

[jira] [Comment Edited] (SPARK-19293) Spark 2.1.x unstable with spark.speculation=true

2017-05-25 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024684#comment-16024684 ] Damian Momot edited comment on SPARK-19293 at 5/25/17 12:50 PM: ---

[jira] [Resolved] (SPARK-20768) PySpark FPGrowth does not expose numPartitions (expert) param

2017-05-25 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-20768. - Resolution: Fixed Fix Version/s: 2.2.0 > PySpark FPGrowth does not expose numPartitions (e

[jira] [Updated] (SPARK-20799) Unable to infer schema for ORC on S3N when secrets are in the URL

2017-05-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-20799: --- Priority: Minor (was: Major) > Unable to infer schema for ORC on S3N when secrets are in the

[jira] [Commented] (SPARK-20879) Spark SQL will read Date type column from avro file as Int

2017-05-25 Thread bing huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024799#comment-16024799 ] bing huang commented on SPARK-20879: Thanks for reply Is there any link to the issue

[jira] [Commented] (SPARK-20879) Spark SQL will read Date type column from avro file as Int

2017-05-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024805#comment-16024805 ] Hyukjin Kwon commented on SPARK-20879: -- Not sure. Probably, https://github.com/datab

[jira] [Commented] (SPARK-20879) Spark SQL will read Date type column from avro file as Int

2017-05-25 Thread bing huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024806#comment-16024806 ] bing huang commented on SPARK-20879: Appreciate, thanks! > Spark SQL will read Date

[jira] [Created] (SPARK-20885) JDBC predicate pushdown uses hardcoded date format

2017-05-25 Thread Peter Halverson (JIRA)
Peter Halverson created SPARK-20885: --- Summary: JDBC predicate pushdown uses hardcoded date format Key: SPARK-20885 URL: https://issues.apache.org/jira/browse/SPARK-20885 Project: Spark Issu

[jira] [Created] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-20886: -- Summary: HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null Key: SPARK-20886 URL: https://issues.apache.org/jira/browse/SPARK-20886

[jira] [Commented] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024885#comment-16024885 ] Steve Loughran commented on SPARK-20886: Stack trace: before {code} Driver stackt

[jira] [Commented] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024886#comment-16024886 ] Steve Loughran commented on SPARK-20886: Stack trace: after {code} 2017-05-25 16:

[jira] [Created] (SPARK-20887) support alternative keys in ConfigBuilder

2017-05-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-20887: --- Summary: support alternative keys in ConfigBuilder Key: SPARK-20887 URL: https://issues.apache.org/jira/browse/SPARK-20887 Project: Spark Issue Type: Improveme

[jira] [Commented] (SPARK-20887) support alternative keys in ConfigBuilder

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024912#comment-16024912 ] Apache Spark commented on SPARK-20887: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-20887) support alternative keys in ConfigBuilder

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20887: Assignee: Apache Spark (was: Wenchen Fan) > support alternative keys in ConfigBuilder > -

[jira] [Assigned] (SPARK-20887) support alternative keys in ConfigBuilder

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20887: Assignee: Wenchen Fan (was: Apache Spark) > support alternative keys in ConfigBuilder > -

[jira] [Commented] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024914#comment-16024914 ] Apache Spark commented on SPARK-20886: -- User 'steveloughran' has created a pull requ

[jira] [Assigned] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20886: Assignee: (was: Apache Spark) > HadoopMapReduceCommitProtocol to fail with message if

[jira] [Assigned] (SPARK-20886) HadoopMapReduceCommitProtocol to fail with message if FileOutputCommitter.getWorkPath==null

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20886: Assignee: Apache Spark > HadoopMapReduceCommitProtocol to fail with message if > FileOutp

[jira] [Resolved] (SPARK-20741) SparkSubmit does not clean up after uploading spark_libs to the distributed cache

2017-05-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20741. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17986 [https://github.co

[jira] [Assigned] (SPARK-20741) SparkSubmit does not clean up after uploading spark_libs to the distributed cache

2017-05-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20741: - Assignee: Lior Regev Issue Type: Improvement (was: Bug) > SparkSubmit does not clean up a

[jira] [Commented] (SPARK-20843) Cannot gracefully kill drivers which take longer than 10 seconds to die

2017-05-25 Thread Michael Allman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025036#comment-16025036 ] Michael Allman commented on SPARK-20843: [~rxin] I'd like to bump this to "Critic

[jira] [Commented] (SPARK-20843) Cannot gracefully kill drivers which take longer than 10 seconds to die

2017-05-25 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025042#comment-16025042 ] Reynold Xin commented on SPARK-20843: - cc [~joshrosen] and [~marmbrus] > Cannot grac

[jira] [Created] (SPARK-20888) Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes

2017-05-25 Thread Michael Allman (JIRA)
Michael Allman created SPARK-20888: -- Summary: Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes Key: SPARK-20888 URL: https://issues.apache.org/jira/browse/SPARK-20888

[jira] [Commented] (SPARK-20888) Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes

2017-05-25 Thread Michael Allman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025059#comment-16025059 ] Michael Allman commented on SPARK-20888: I will work on a PR for this and try to

[jira] [Created] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Wayne Zhang (JIRA)
Wayne Zhang created SPARK-20889: --- Summary: SparkR grouped documentation for Column methods Key: SPARK-20889 URL: https://issues.apache.org/jira/browse/SPARK-20889 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025062#comment-16025062 ] Apache Spark commented on SPARK-20889: -- User 'actuaryzhang' has created a pull reque

[jira] [Assigned] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20889: Assignee: (was: Apache Spark) > SparkR grouped documentation for Column methods >

[jira] [Assigned] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20889: Assignee: Apache Spark > SparkR grouped documentation for Column methods > ---

[jira] [Updated] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-20889: - Labels: documentation (was: ) > SparkR grouped documentation for Column methods > --

[jira] [Updated] (SPARK-20889) SparkR grouped documentation for Column methods

2017-05-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-20889: - Issue Type: Documentation (was: Improvement) > SparkR grouped documentation for Column methods >

[jira] [Updated] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

2017-05-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20874: - Fix Version/s: 2.2.0 2.1.2 > The "examples" project doesn't depend on Structur

[jira] [Updated] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

2017-05-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20874: - Affects Version/s: 2.2.0 > The "examples" project doesn't depend on Structured Streaming Kafka so

[jira] [Resolved] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

2017-05-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20874. -- Resolution: Fixed > The "examples" project doesn't depend on Structured Streaming Kafka source

[jira] [Commented] (SPARK-20888) Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025112#comment-16025112 ] Apache Spark commented on SPARK-20888: -- User 'mallman' has created a pull request fo

[jira] [Assigned] (SPARK-20888) Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20888: Assignee: Apache Spark > Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark S

[jira] [Assigned] (SPARK-20888) Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE in Spark SQL 2.1 to 2.2 migration notes

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20888: Assignee: (was: Apache Spark) > Document HiveCaseSensitiveInferenceMode.INFER_AND_SAVE

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Description: This bug is like https://issues.apache.org/jira/browse/SPARK-19300. but I have updated my

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025227#comment-16025227 ] Shixiong Zhu commented on SPARK-20882: -- [~cenyuhai] are you using shuffle service?

[jira] [Updated] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-20882: -- Description: This bug is like https://issues.apache.org/jira/browse/SPARK-19300. but I have updated my

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

2017-05-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025377#comment-16025377 ] Shixiong Zhu commented on SPARK-20882: -- NVM. I saw NettyBlockTransferService in the

[jira] [Created] (SPARK-20890) Add min and max functions for dataset aggregation

2017-05-25 Thread Ruben Janssen (JIRA)
Ruben Janssen created SPARK-20890: - Summary: Add min and max functions for dataset aggregation Key: SPARK-20890 URL: https://issues.apache.org/jira/browse/SPARK-20890 Project: Spark Issue Typ

[jira] [Commented] (SPARK-20890) Add min and max functions for dataset aggregation

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025407#comment-16025407 ] Apache Spark commented on SPARK-20890: -- User 'setjet' has created a pull request for

[jira] [Assigned] (SPARK-20890) Add min and max functions for dataset aggregation

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20890: Assignee: (was: Apache Spark) > Add min and max functions for dataset aggregation > --

[jira] [Assigned] (SPARK-20890) Add min and max functions for dataset aggregation

2017-05-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20890: Assignee: Apache Spark > Add min and max functions for dataset aggregation > -

[jira] [Created] (SPARK-20891) Reduce duplicate code in typedaggregators.scala

2017-05-25 Thread Ruben Janssen (JIRA)
Ruben Janssen created SPARK-20891: - Summary: Reduce duplicate code in typedaggregators.scala Key: SPARK-20891 URL: https://issues.apache.org/jira/browse/SPARK-20891 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20891) Reduce duplicate code in typedaggregators.scala

2017-05-25 Thread Ruben Janssen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025412#comment-16025412 ] Ruben Janssen commented on SPARK-20891: --- Working on this, waiting for SPARK-20890 t

[jira] [Comment Edited] (SPARK-20411) New features for expression.scalalang.typed

2017-05-25 Thread Ruben Janssen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025416#comment-16025416 ] Ruben Janssen edited comment on SPARK-20411 at 5/25/17 9:19 PM: ---

  1   2   >