[jira] [Resolved] (SPARK-14767) Codegen "no constructor found" errors with Maps inside case classes in Datasets

2016-12-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-14767. - Resolution: Fixed Fix Version/s: 2.2.0 > Codegen "no constructor found" errors with Maps

[jira] [Commented] (SPARK-4591) Algorithm/model parity for spark.ml (Scala)

2016-12-12 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744481#comment-15744481 ] Felix Cheung commented on SPARK-4591: - Is SVM part of this? > Algorithm/model parity for spark.ml

[jira] [Resolved] (SPARK-18717) Datasets - crash (compile exception) when mapping to immutable scala map

2016-12-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18717. - Resolution: Fixed Assignee: Andrew Ray Fix Version/s: 2.2.0 > Datasets - crash

[jira] [Commented] (SPARK-18823) Assignation by column name variable not available or bug?

2016-12-12 Thread Vicente Masip (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744465#comment-15744465 ] Vicente Masip commented on SPARK-18823: --- Well maybe I haven't explained myself. I wrote the right

[jira] [Commented] (SPARK-18829) Printing to logger

2016-12-12 Thread David Hodeffi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744464#comment-15744464 ] David Hodeffi commented on SPARK-18829: --- The issue is it's hard to debug application on yarn.

[jira] [Updated] (SPARK-18838) High latency of event processing for large jobs

2016-12-12 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sital Kedia updated SPARK-18838: Description: Currently we are observing the issue of very high event processing delay in driver's

[jira] [Commented] (SPARK-18823) Assignation by column name variable not available or bug?

2016-12-12 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744415#comment-15744415 ] Felix Cheung commented on SPARK-18823: -- How important it is to support df[[myname]] <- c(1:nrow(df))

[jira] [Commented] (SPARK-18794) SparkR vignette update: gbt

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744403#comment-15744403 ] Apache Spark commented on SPARK-18794: -- User 'mengxr' has created a pull request for this issue:

[jira] [Commented] (SPARK-18793) SparkR vignette update: random forest

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744390#comment-15744390 ] Apache Spark commented on SPARK-18793: -- User 'mengxr' has created a pull request for this issue:

[jira] [Commented] (SPARK-18792) SparkR vignette update: logit

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744387#comment-15744387 ] Apache Spark commented on SPARK-18792: -- User 'mengxr' has created a pull request for this issue:

[jira] [Commented] (SPARK-17147) Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2016-12-12 Thread caolan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744380#comment-15744380 ] caolan commented on SPARK-17147: Sure, I will give a try on your branch, will let you know the result. >

[jira] [Updated] (SPARK-18840) HDFSCredentialProvider throws exception in non-HDFS security environment

2016-12-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-18840: Priority: Minor (was: Major) > HDFSCredentialProvider throws exception in non-HDFS security

[jira] [Commented] (SPARK-18840) HDFSCredentialProvider throws exception in non-HDFS security environment

2016-12-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744370#comment-15744370 ] Saisai Shao commented on SPARK-18840: - This problem also existed in branch 1.6, but the fix is a

[jira] [Created] (SPARK-18840) HDFSCredentialProvider throws exception in non-HDFS security environment

2016-12-12 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-18840: --- Summary: HDFSCredentialProvider throws exception in non-HDFS security environment Key: SPARK-18840 URL: https://issues.apache.org/jira/browse/SPARK-18840 Project:

[jira] [Updated] (SPARK-18795) SparkR vignette update: ksTest

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-18795: -- Assignee: Miao Wang (was: Xiangrui Meng) > SparkR vignette update: ksTest >

[jira] [Assigned] (SPARK-18793) SparkR vignette update: random forest

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-18793: - Assignee: Xiangrui Meng > SparkR vignette update: random forest >

[jira] [Commented] (SPARK-18795) SparkR vignette update: ksTest

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744359#comment-15744359 ] Xiangrui Meng commented on SPARK-18795: --- [~wangmiao1981] Any updates? > SparkR vignette update:

[jira] [Assigned] (SPARK-18794) SparkR vignette update: gbt

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-18794: - Assignee: Xiangrui Meng > SparkR vignette update: gbt > --- > >

[jira] [Assigned] (SPARK-18795) SparkR vignette update: ksTest

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-18795: - Assignee: Xiangrui Meng > SparkR vignette update: ksTest >

[jira] [Resolved] (SPARK-18797) Update spark.logit in sparkr-vignettes

2016-12-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-18797. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue resolved by

[jira] [Created] (SPARK-18839) Executor is active on web, but actually is dead

2016-12-12 Thread meiyoula (JIRA)
meiyoula created SPARK-18839: Summary: Executor is active on web, but actually is dead Key: SPARK-18839 URL: https://issues.apache.org/jira/browse/SPARK-18839 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-12 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-18796. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 16220

[jira] [Comment Edited] (SPARK-18804) Join doesn't work in Spark on Bigger tables

2016-12-12 Thread Gopal Nagar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744246#comment-15744246 ] Gopal Nagar edited comment on SPARK-18804 at 12/13/16 6:00 AM: --- Apologies

[jira] [Commented] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744271#comment-15744271 ] Apache Spark commented on SPARK-18281: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18281: Assignee: (was: Apache Spark) > toLocalIterator yields time out error on pyspark2 >

[jira] [Assigned] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18281: Assignee: Apache Spark > toLocalIterator yields time out error on pyspark2 >

[jira] [Reopened] (SPARK-18804) Join doesn't work in Spark on Bigger tables

2016-12-12 Thread Gopal Nagar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Nagar reopened SPARK-18804: - Apologies for marking this JIRA as bug. This may not be a bug in Spark. But i wanted to get some

[jira] [Updated] (SPARK-18804) Join doesn't work in Spark on Bigger tables

2016-12-12 Thread Gopal Nagar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Nagar updated SPARK-18804: Issue Type: Question (was: Brainstorming) > Join doesn't work in Spark on Bigger tables >

[jira] [Updated] (SPARK-18804) Join doesn't work in Spark on Bigger tables

2016-12-12 Thread Gopal Nagar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Nagar updated SPARK-18804: Issue Type: Brainstorming (was: Bug) > Join doesn't work in Spark on Bigger tables >

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2016-12-12 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744086#comment-15744086 ] Sital Kedia commented on SPARK-18838: - [~rxin], [~zsxwing] - Any thoughts on this? > High latency

[jira] [Updated] (SPARK-18838) High latency of event processing for large jobs

2016-12-12 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sital Kedia updated SPARK-18838: Description: Currently we are observing the issue of very high event processing delay in driver's

[jira] [Created] (SPARK-18838) High latency of event processing for large jobs

2016-12-12 Thread Sital Kedia (JIRA)
Sital Kedia created SPARK-18838: --- Summary: High latency of event processing for large jobs Key: SPARK-18838 URL: https://issues.apache.org/jira/browse/SPARK-18838 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744047#comment-15744047 ] Liang-Chi Hsieh commented on SPARK-18281: - [~mwdus...@us.ibm.com] I can reproduce your issue. I

[jira] [Commented] (SPARK-17932) Failed to run SQL "show table extended like table_name" in Spark2.0.0

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744035#comment-15744035 ] Apache Spark commented on SPARK-17932: -- User 'jiangxb1987' has created a pull request for this

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Description: *previous*: !ui-2.0.0.gif! *current*: !ui-2.1.0.gif! was:!attached-image.gif!

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Description: !attached-image.gif! > It will not hidden if job or stage description too long >

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Attachment: ui-2.0.0.gif ui-2.1.0.gif > It will not hidden if job or stage

[jira] [Created] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18837: --- Summary: It will not hidden if job or stage description too long Key: SPARK-18837 URL: https://issues.apache.org/jira/browse/SPARK-18837 Project: Spark Issue

[jira] [Updated] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-18113: --- Description: Executor sends *AskPermissionToCommitOutput* to driver failed, and retry another

[jira] [Updated] (SPARK-17664) Failed to saveAsHadoop when speculate is enabled

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-17664: --- Description: >From follow logs, task 22 has failed 4 times because of "the driver did not

[jira] [Updated] (SPARK-18834) Expose event time time stats through StreamingQueryProgress

2016-12-12 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-18834: -- Summary: Expose event time time stats through StreamingQueryProgress (was: Expose event time

[jira] [Updated] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-18816: - Assignee: Alex Bozarth > executor page fails to show log links if executors are added after an app is >

[jira] [Updated] (SPARK-18023) Adam optimizer

2016-12-12 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18023: -- Shepherd: (was: Joseph K. Bradley) > Adam optimizer > -- > >

[jira] [Commented] (SPARK-18823) Assignation by column name variable not available or bug?

2016-12-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743648#comment-15743648 ] Shivaram Venkataraman commented on SPARK-18823: --- We don't support assigning to columns

[jira] [Updated] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

2016-12-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18773: - Assignee: Marcelo Vanzin > Make translation of Spark configs to commons-crypto configs

[jira] [Resolved] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

2016-12-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18773. -- Resolution: Fixed > Make translation of Spark configs to commons-crypto configs consistent >

[jira] [Updated] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

2016-12-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18773: - Fix Version/s: 2.2.0 > Make translation of Spark configs to commons-crypto configs consistent >

[jira] [Assigned] (SPARK-18836) Serialize Task Metrics once per stage

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18836: Assignee: Apache Spark > Serialize Task Metrics once per stage >

[jira] [Commented] (SPARK-18836) Serialize Task Metrics once per stage

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743623#comment-15743623 ] Apache Spark commented on SPARK-18836: -- User 'shivaram' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18836) Serialize Task Metrics once per stage

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18836: Assignee: (was: Apache Spark) > Serialize Task Metrics once per stage >

[jira] [Created] (SPARK-18836) Serialize Task Metrics once per stage

2016-12-12 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-18836: - Summary: Serialize Task Metrics once per stage Key: SPARK-18836 URL: https://issues.apache.org/jira/browse/SPARK-18836 Project: Spark

[jira] [Assigned] (SPARK-18835) Do not expose shaded types in JavaTypeInference API

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18835: Assignee: (was: Apache Spark) > Do not expose shaded types in JavaTypeInference API >

[jira] [Assigned] (SPARK-18835) Do not expose shaded types in JavaTypeInference API

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18835: Assignee: Apache Spark > Do not expose shaded types in JavaTypeInference API >

[jira] [Commented] (SPARK-18835) Do not expose shaded types in JavaTypeInference API

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743565#comment-15743565 ] Apache Spark commented on SPARK-18835: -- User 'vanzin' has created a pull request for this issue:

[jira] [Created] (SPARK-18835) Do not expose shaded types in JavaTypeInference API

2016-12-12 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-18835: -- Summary: Do not expose shaded types in JavaTypeInference API Key: SPARK-18835 URL: https://issues.apache.org/jira/browse/SPARK-18835 Project: Spark

[jira] [Assigned] (SPARK-18834) Expose event time and processing time stats through StreamingQueryProgress

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18834: Assignee: Apache Spark (was: Tathagata Das) > Expose event time and processing time

[jira] [Assigned] (SPARK-18834) Expose event time and processing time stats through StreamingQueryProgress

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18834: Assignee: Tathagata Das (was: Apache Spark) > Expose event time and processing time

[jira] [Commented] (SPARK-18834) Expose event time and processing time stats through StreamingQueryProgress

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743528#comment-15743528 ] Apache Spark commented on SPARK-18834: -- User 'tdas' has created a pull request for this issue:

[jira] [Commented] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-12 Thread Mike Dusenberry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743524#comment-15743524 ] Mike Dusenberry commented on SPARK-18281: - I'm also seeing the same error with both Python 2.7

[jira] [Commented] (SPARK-18813) MLlib 2.2 Roadmap

2016-12-12 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743513#comment-15743513 ] Joseph K. Bradley commented on SPARK-18813: --- Those are definitely some top items in my mind

[jira] [Created] (SPARK-18834) Expose event time and processing time stats through StreamingQueryProgress

2016-12-12 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-18834: - Summary: Expose event time and processing time stats through StreamingQueryProgress Key: SPARK-18834 URL: https://issues.apache.org/jira/browse/SPARK-18834

[jira] [Updated] (SPARK-18834) Expose event time and processing time stats through StreamingQueryProgress

2016-12-12 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-18834: -- Issue Type: Improvement (was: Bug) > Expose event time and processing time stats through

[jira] [Commented] (SPARK-18752) "isSrcLocal" parameter to Hive loadTable / loadPartition should come from user

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743486#comment-15743486 ] Apache Spark commented on SPARK-18752: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18816: Assignee: Apache Spark > executor page fails to show log links if executors are added

[jira] [Created] (SPARK-18833) Changing partition location using the 'ALTER TABLE .. SET LOCATION' command via beeline doesn't get reflected in Spark

2016-12-12 Thread Salil Surendran (JIRA)
Salil Surendran created SPARK-18833: --- Summary: Changing partition location using the 'ALTER TABLE .. SET LOCATION' command via beeline doesn't get reflected in Spark Key: SPARK-18833 URL:

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743453#comment-15743453 ] Apache Spark commented on SPARK-18816: -- User 'ajbozarth' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18816: Assignee: (was: Apache Spark) > executor page fails to show log links if executors

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743419#comment-15743419 ] Alex Bozarth commented on SPARK-18816: -- I have a fix, just running a few tests then I'll open a pr

[jira] [Resolved] (SPARK-18810) SparkR install.spark does not work for RCs, snapshots

2016-12-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-18810. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue

[jira] [Resolved] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-12 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18681. --- Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 2.1.1 > Throw

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743362#comment-15743362 ] Reynold Xin commented on SPARK-18676: - Can we just increase the size by 5X if it is a Parquet or ORC

[jira] [Updated] (SPARK-18746) Add implicit encoders for BigDecimal, timestamp and date

2016-12-12 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SPARK-18746: - Description: Run the code below in spark-shell, there will be an error: {code} scala>

[jira] [Updated] (SPARK-18746) Add implicit encoders for BigDecimal, timestamp and date

2016-12-12 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SPARK-18746: - Description: Run the code below in spark-shell, there will be an error: {code} scala>

[jira] [Updated] (SPARK-18746) Add implicit encoders for BigDecimal, timestamp and date

2016-12-12 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SPARK-18746: - Description: Run the code below in spark-shell, there will be an error: {code} scala>

[jira] [Updated] (SPARK-18746) Add implicit encoders for BigDecimal, timestamp and date

2016-12-12 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SPARK-18746: - Summary: Add implicit encoders for BigDecimal, timestamp and date (was: Add

[jira] [Resolved] (SPARK-18752) "isSrcLocal" parameter to Hive loadTable / loadPartition should come from user

2016-12-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-18752. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16179

[jira] [Commented] (SPARK-18832) Spark SQL: Incorrect error message on calling registered UDF.

2016-12-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743324#comment-15743324 ] Dongjoon Hyun commented on SPARK-18832: --- Hi, [~roadster11]. Could you confirm that your Hive UDTF

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743313#comment-15743313 ] Alex Bozarth commented on SPARK-18816: -- And I still don't think this is a blocker, but I respect

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743307#comment-15743307 ] Alex Bozarth commented on SPARK-18816: -- I'm actually looking into a bit right now and I think it's

[jira] [Resolved] (SPARK-18765) Make values for spark.yarn.{am|driver|executor}.memoryOverhead have configurable units

2016-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-18765. Resolution: Won't Fix I missed that this is already fixed in 2.0; since it's a new

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743298#comment-15743298 ] Yin Huai commented on SPARK-18816: -- Yea, log pages are still there. But, without those links on the

[jira] [Updated] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-18816: - Priority: Blocker (was: Major) > executor page fails to show log links if executors are added after an

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743288#comment-15743288 ] Alex Bozarth commented on SPARK-18816: -- Thanks for following up, I was able to recreate the issue,

[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched

2016-12-12 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743260#comment-15743260 ] Yin Huai commented on SPARK-18816: -- [~ajbozarth] Yea, please take a look. Thanks! The reasons that I

[jira] [Updated] (SPARK-14519) Cross-publish Kafka for Scala 2.12

2016-12-12 Thread Jakob Odersky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Odersky updated SPARK-14519: -- Summary: Cross-publish Kafka for Scala 2.12 (was: Cross-publish Kafka for Scala 2.12.0-M4) >

[jira] [Resolved] (SPARK-16297) Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc dialect

2016-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-16297. Resolution: Fixed Assignee: Oussama Mekni Fix Version/s: 2.2.0 > Mapping

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-12 Thread Michael Allman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743076#comment-15743076 ] Michael Allman commented on SPARK-18676: I'm sorry I have not had time to provide more

[jira] [Resolved] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15844. Resolution: Fixed Assignee: Steve Loughran Fix Version/s: 2.2.0 >

[jira] [Updated] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default

2016-12-12 Thread Brendan Dwyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brendan Dwyer updated SPARK-18817: -- Description: Per CRAN policies https://cran.r-project.org/web/packages/policies.html {quote} -

[jira] [Closed] (SPARK-14631) "drop database cascade" needs to unregister functions for HiveExternalCatalog

2016-12-12 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Wang closed SPARK-14631. --- Resolution: Not A Problem > "drop database cascade" needs to unregister functions for

[jira] [Commented] (SPARK-17147) Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2016-12-12 Thread Sean McKibben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743024#comment-15743024 ] Sean McKibben commented on SPARK-17147: --- I apologize for the extended radio silence on this. I've

[jira] [Created] (SPARK-18832) Spark SQL: Incorrect error message on calling registered UDF.

2016-12-12 Thread Lokesh Yadav (JIRA)
Lokesh Yadav created SPARK-18832: Summary: Spark SQL: Incorrect error message on calling registered UDF. Key: SPARK-18832 URL: https://issues.apache.org/jira/browse/SPARK-18832 Project: Spark

[jira] [Commented] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

2016-12-12 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742879#comment-15742879 ] Nattavut Sutyanyong commented on SPARK-18814: - q92 has the same pattern as q32 and my

[jira] [Commented] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

2016-12-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742828#comment-15742828 ] Kazuaki Ishizaki commented on SPARK-18814: -- I found the same error

[jira] [Commented] (SPARK-18829) Printing to logger

2016-12-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742806#comment-15742806 ] Sean Owen commented on SPARK-18829: --- If the request is "return the explain output as a string instead",

[jira] [Commented] (SPARK-18795) SparkR vignette update: ksTest

2016-12-12 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742790#comment-15742790 ] Joseph K. Bradley commented on SPARK-18795: --- OK thanks! Let's keep these vignettes short to

[jira] [Updated] (SPARK-18325) SparkR 2.1 QA: Check for new R APIs requiring example code

2016-12-12 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18325: -- Fix Version/s: 2.2.0 > SparkR 2.1 QA: Check for new R APIs requiring example code >

[jira] [Commented] (SPARK-17409) Query in CTAS is Optimized Twice

2016-12-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742691#comment-15742691 ] Xiao Li commented on SPARK-17409: - Sure, will follow it. > Query in CTAS is Optimized Twice >

[jira] [Commented] (SPARK-18825) Eliminate duplicate links in SparkR API doc index

2016-12-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742601#comment-15742601 ] Shivaram Venkataraman commented on SPARK-18825: --- Yeah this is tricky because of CRAN

[jira] [Commented] (SPARK-17892) Query in CTAS is Optimized Twice (branch-2.0)

2016-12-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742575#comment-15742575 ] Josh Rosen commented on SPARK-17892: In the future, please re-use the existing JIRA when backporting

  1   2   >