[jira] [Updated] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16621: --- Assignee: Dongjoon Hyun > Generate stable SQLs in SQLBuilder > -- >

[jira] [Resolved] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16621. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14257

[jira] [Commented] (SPARK-16748) Errors thrown by UDFs cause TreeNodeException when the query has an ORDER BY clause

2016-07-26 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395063#comment-15395063 ] Yin Huai commented on SPARK-16748: -- Seems we should should not wrap the NPE in a TreeNodeException. >

[jira] [Updated] (SPARK-16748) Errors thrown by UDFs cause TreeNodeException when the query has an ORDER BY clause

2016-07-26 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-16748: - Description: {code} import org.apache.spark.sql.functions._ val myUDF = udf((c: String) =>

[jira] [Created] (SPARK-16748) Errors thrown by UDFs cause TreeNodeException when the query has an ORDER BY clause

2016-07-26 Thread Yin Huai (JIRA)
Yin Huai created SPARK-16748: Summary: Errors thrown by UDFs cause TreeNodeException when the query has an ORDER BY clause Key: SPARK-16748 URL: https://issues.apache.org/jira/browse/SPARK-16748 Project:

[jira] [Updated] (SPARK-16734) Make sure examples in all language bindings are consistent

2016-07-26 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-16734: - Target Version/s: 2.0.1, 2.1.0 (was: 2.0.0, 2.0.1, 2.1.0) > Make sure examples in all language bindings

[jira] [Closed] (SPARK-16747) MQTT as a streaming source for SQL Streaming.

2016-07-26 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma closed SPARK-16747. --- Resolution: Won't Fix > MQTT as a streaming source for SQL Streaming. >

[jira] [Comment Edited] (SPARK-16747) MQTT as a streaming source for SQL Streaming.

2016-07-26 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395046#comment-15395046 ] Prashant Sharma edited comment on SPARK-16747 at 7/27/16 5:05 AM: -- This

[jira] [Commented] (SPARK-16747) MQTT as a streaming source for SQL Streaming.

2016-07-26 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395046#comment-15395046 ] Prashant Sharma commented on SPARK-16747: - This created by mistake. > MQTT as a streaming source

[jira] [Created] (SPARK-16747) MQTT as a streaming source for SQL Streaming.

2016-07-26 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-16747: --- Summary: MQTT as a streaming source for SQL Streaming. Key: SPARK-16747 URL: https://issues.apache.org/jira/browse/SPARK-16747 Project: Spark Issue

[jira] [Commented] (SPARK-15194) Add Python ML API for MultivariateGaussian

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395018#comment-15395018 ] Apache Spark commented on SPARK-15194: -- User 'praveendareddy21' has created a pull request for this

[jira] [Updated] (SPARK-16746) Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout

2016-07-26 Thread Hongyao Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyao Zhao updated SPARK-16746: - Description: I wrote a spark streaming program which consume 1000 messages from one topic of

[jira] [Updated] (SPARK-16746) Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout

2016-07-26 Thread Hongyao Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyao Zhao updated SPARK-16746: - Description: I wrote a spark streaming program which consume 1000 messages from one topic of

[jira] [Updated] (SPARK-16746) Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout

2016-07-26 Thread Hongyao Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyao Zhao updated SPARK-16746: - Description: I wrote a spark streaming program which consume 1000 messages from one topic of

[jira] [Updated] (SPARK-16746) Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout

2016-07-26 Thread Hongyao Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyao Zhao updated SPARK-16746: - Description: I wrote a spark streaming program which consume 1000 messages from one topic of

[jira] [Created] (SPARK-16746) Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout

2016-07-26 Thread Hongyao Zhao (JIRA)
Hongyao Zhao created SPARK-16746: Summary: Spark streaming lost data when ReceiverTracker writes Blockinfo to hdfs timeout Key: SPARK-16746 URL: https://issues.apache.org/jira/browse/SPARK-16746

[jira] [Commented] (SPARK-16446) Gaussian Mixture Model wrapper in SparkR

2016-07-26 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395003#comment-15395003 ] Yanbo Liang commented on SPARK-16446: - Will send PR soon. > Gaussian Mixture Model wrapper in SparkR

[jira] [Created] (SPARK-16745) Spark job completed however have to wait for 13 mins (data size is small)

2016-07-26 Thread Joe Chong (JIRA)
Joe Chong created SPARK-16745: - Summary: Spark job completed however have to wait for 13 mins (data size is small) Key: SPARK-16745 URL: https://issues.apache.org/jira/browse/SPARK-16745 Project: Spark

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only

[jira] [Resolved] (SPARK-16524) Add RowBatch and RowBasedHashMapGenerator

2016-07-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-16524. - Resolution: Fixed Assignee: Qifan Pu Fix Version/s: 2.1.0 > Add RowBatch and

[jira] [Comment Edited] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394875#comment-15394875 ] Marcelo Vanzin edited comment on SPARK-16725 at 7/27/16 1:04 AM: - There

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394875#comment-15394875 ] Marcelo Vanzin commented on SPARK-16725: There is a clear guideline for when you need a library

[jira] [Commented] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Liang Ke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394864#comment-15394864 ] Liang Ke commented on SPARK-16735: -- https://github.com/apache/spark/pull/14374 > Fail to create a map

[jira] [Commented] (SPARK-16715) Fix a potential ExprId conflict for SubexpressionEliminationSuite."Semantic equals and hash"

2016-07-26 Thread Liang Ke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394860#comment-15394860 ] Liang Ke commented on SPARK-16715: -- Sorry, mistake the jira id. > Fix a potential ExprId conflict for

[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended

2016-07-26 Thread Kevin Grealish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394812#comment-15394812 ] Kevin Grealish commented on SPARK-16744: Issue discovered when doing narrower fix for SPARK-16110

[jira] [Created] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended

2016-07-26 Thread Kevin Grealish (JIRA)
Kevin Grealish created SPARK-16744: -- Summary: spark.yarn.appMasterEnv handling assumes values should be appended Key: SPARK-16744 URL: https://issues.apache.org/jira/browse/SPARK-16744 Project:

[jira] [Assigned] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16735: Assignee: Apache Spark > Fail to create a map contains decimal type with literals having

[jira] [Assigned] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16735: Assignee: (was: Apache Spark) > Fail to create a map contains decimal type with

[jira] [Commented] (SPARK-16715) Fix a potential ExprId conflict for SubexpressionEliminationSuite."Semantic equals and hash"

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394783#comment-15394783 ] Apache Spark commented on SPARK-16715: -- User 'biglobster' has created a pull request for this issue:

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394768#comment-15394768 ] Min Wei commented on SPARK-16725: - Still worth to use a newer version of Guava. Looks like the issue of

[jira] [Updated] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16621: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-16576 > Generate stable SQLs in

[jira] [Commented] (SPARK-15232) Add subquery SQL building tests to LogicalPlanToSQLSuite

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394738#comment-15394738 ] Dongjoon Hyun commented on SPARK-15232: --- Hi, [~hvanhovell]. May I work on this issue? > Add

[jira] [Issue Comment Deleted] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Liang Ke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Ke updated SPARK-16735: - Comment: was deleted (was: hi, if some one can help me to push my patch to github ? thx a lot : )) >

[jira] [Updated] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Liang Ke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Ke updated SPARK-16735: - Attachment: (was: SPARK-16735.patch) > Fail to create a map contains decimal type with literals

[jira] [Comment Edited] (SPARK-16445) Multilayer Perceptron Classifier wrapper in SparkR

2016-07-26 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394654#comment-15394654 ] Xin Ren edited comment on SPARK-16445 at 7/26/16 10:17 PM: --- I'm still working

[jira] [Commented] (SPARK-16445) Multilayer Perceptron Classifier wrapper in SparkR

2016-07-26 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394654#comment-15394654 ] Xin Ren commented on SPARK-16445: - I'm still working on it, hopefully by end of this weekend I can submit

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394652#comment-15394652 ] Sean Owen commented on SPARK-16725: --- Again there are two Guavas here: the unshaded one for Hadoop et

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Sylvain Zimmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394644#comment-15394644 ] Sylvain Zimmer commented on SPARK-16740: OK! Looks like that would be [~davies] :-) >

[jira] [Commented] (SPARK-16445) Multilayer Perceptron Classifier wrapper in SparkR

2016-07-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394642#comment-15394642 ] Xiangrui Meng commented on SPARK-16445: --- [~iamshrek] Any updates? > Multilayer Perceptron

[jira] [Commented] (SPARK-16446) Gaussian Mixture Model wrapper in SparkR

2016-07-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394639#comment-15394639 ] Xiangrui Meng commented on SPARK-16446: --- [~yanboliang] Any updates? > Gaussian Mixture Model

[jira] [Updated] (SPARK-16444) Isotonic Regression wrapper in SparkR

2016-07-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-16444: -- Shepherd: Junyang Qian > Isotonic Regression wrapper in SparkR >

[jira] [Comment Edited] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394634#comment-15394634 ] Dongjoon Hyun edited comment on SPARK-16740 at 7/26/16 10:03 PM: - You had

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394634#comment-15394634 ] Dongjoon Hyun commented on SPARK-16740: --- You had better look up the one you made that with `git

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Sylvain Zimmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394628#comment-15394628 ] Sylvain Zimmer commented on SPARK-16740: Thanks! I just did. Let me know if that's okay. >

[jira] [Assigned] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16740: Assignee: Apache Spark > joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

[jira] [Assigned] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16740: Assignee: (was: Apache Spark) > joins.LongToUnsafeRowMap crashes with

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394624#comment-15394624 ] Apache Spark commented on SPARK-16740: -- User 'sylvinus' has created a pull request for this issue:

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394622#comment-15394622 ] Min Wei commented on SPARK-16725: - I would love to see Spark successful as a platform, and not caught up

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394587#comment-15394587 ] Sean Owen commented on SPARK-16725: --- Thanks for the reminder [~vanzin], I caught up again on the

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394572#comment-15394572 ] Min Wei commented on SPARK-16725: - First I am not blocked. Plus I can hack anything I need to make it

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394555#comment-15394555 ] Dongjoon Hyun commented on SPARK-16740: --- Maybe, you will modify the following line? {code} val

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394553#comment-15394553 ] Dongjoon Hyun commented on SPARK-16740: --- Hi, [~sylvinus]. It looks like that. Could you make a PR

[jira] [Created] (SPARK-16743) converter and access code out of sync: createDataFrame on RDD[Option[C]] fails with MatchError

2016-07-26 Thread Daniel Barclay (JIRA)
Daniel Barclay created SPARK-16743: -- Summary: converter and access code out of sync: createDataFrame on RDD[Option[C]] fails with MatchError Key: SPARK-16743 URL:

[jira] [Created] (SPARK-16742) Kerberos support for Spark on Mesos

2016-07-26 Thread Michael Gummelt (JIRA)
Michael Gummelt created SPARK-16742: --- Summary: Kerberos support for Spark on Mesos Key: SPARK-16742 URL: https://issues.apache.org/jira/browse/SPARK-16742 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-16741) spark.speculation causes duplicate rows in df.write.jdbc()

2016-07-26 Thread Zoltan Fedor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394515#comment-15394515 ] Zoltan Fedor commented on SPARK-16741: -- Most databases allow you to set unique keys and disallow the

[jira] [Commented] (SPARK-12261) pyspark crash for large dataset

2016-07-26 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394508#comment-15394508 ] Shea Parkes commented on SPARK-12261: - I still can't get this bug to reproduce reliably locally, but

[jira] [Updated] (SPARK-16741) spark.speculation causes duplicate rows in df.write.jdbc()

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16741: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) Yeah, the problem is that it's

[jira] [Commented] (SPARK-16741) spark.speculation causes duplicate rows in df.write.jdbc()

2016-07-26 Thread Zoltan Fedor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394501#comment-15394501 ] Zoltan Fedor commented on SPARK-16741: -- Thanks Sean. I agree, it is debatable whether this is a bug

[jira] [Commented] (SPARK-16741) spark.speculation causes duplicate rows in df.write.jdbc()

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394480#comment-15394480 ] Sean Owen commented on SPARK-16741: --- Yeah, because the output isn't idempotent, I'm not sure you can

[jira] [Commented] (SPARK-16611) Expose several hidden DataFrame/RDD functions

2016-07-26 Thread Alok Singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394485#comment-15394485 ] Alok Singh commented on SPARK-16611: Hi [~shivaram] Sorry for late reply due to vacation. Here is

[jira] [Created] (SPARK-16741) spark.speculation causes duplicate rows in df.write.jdbc()

2016-07-26 Thread Zoltan Fedor (JIRA)
Zoltan Fedor created SPARK-16741: Summary: spark.speculation causes duplicate rows in df.write.jdbc() Key: SPARK-16741 URL: https://issues.apache.org/jira/browse/SPARK-16741 Project: Spark

[jira] [Commented] (SPARK-8518) Log-linear models for survival analysis

2016-07-26 Thread Eliano Marques (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394450#comment-15394450 ] Eliano Marques commented on SPARK-8518: --- Would it be possible to add some information about the

[jira] [Comment Edited] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Sylvain Zimmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394437#comment-15394437 ] Sylvain Zimmer edited comment on SPARK-16740 at 7/26/16 7:54 PM: - I'm not

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Sylvain Zimmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394437#comment-15394437 ] Sylvain Zimmer commented on SPARK-16740: I'm not a Scala expert but from a quick review of the

[jira] [Created] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-07-26 Thread Sylvain Zimmer (JIRA)
Sylvain Zimmer created SPARK-16740: -- Summary: joins.LongToUnsafeRowMap crashes with NegativeArraySizeException Key: SPARK-16740 URL: https://issues.apache.org/jira/browse/SPARK-16740 Project: Spark

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394329#comment-15394329 ] Marcelo Vanzin commented on SPARK-16725: Again, if you code depends on a specific version of

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394311#comment-15394311 ] Min Wei commented on SPARK-16725: - I am not sure if the Hadoop jar is the issue, as I am just using the

[jira] [Updated] (SPARK-16718) gbm-style treeboost

2016-07-26 Thread Vladimir Feinberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Feinberg updated SPARK-16718: -- Description: .As an initial minimal change, we should provide TreeBoost as implemented

[jira] [Resolved] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15703. --- Resolution: Fixed Fix Version/s: 2.1.0 > Make ListenerBus event queue size

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Priority: Minor (was: Critical) > Make ListenerBus event queue size configurable >

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Issue Type: Improvement (was: Bug) > Make ListenerBus event queue size configurable >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394271#comment-15394271 ] Thomas Graves commented on SPARK-15703: --- Using this jira to make lsitenerbus configurable and we

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Assignee: Dhruve Ashar > Make ListenerBus event queue size configurable >

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Summary: Make ListenerBus event queue size configurable (was: Spark UI doesn't show all tasks

[jira] [Created] (SPARK-16739) GBTClassifier should be a Classifier, not Predictor

2016-07-26 Thread Vladimir Feinberg (JIRA)
Vladimir Feinberg created SPARK-16739: - Summary: GBTClassifier should be a Classifier, not Predictor Key: SPARK-16739 URL: https://issues.apache.org/jira/browse/SPARK-16739 Project: Spark

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394214#comment-15394214 ] Marcelo Vanzin commented on SPARK-16725: Spark depends on Guava 14 and shades it. The Guava you

[jira] [Assigned] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16736: Assignee: (was: Apache Spark) > remove redundant FileSystem status checks calls from

[jira] [Commented] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394196#comment-15394196 ] Apache Spark commented on SPARK-16736: -- User 'steveloughran' has created a pull request for this

[jira] [Assigned] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16736: Assignee: Apache Spark > remove redundant FileSystem status checks calls from Spark

[jira] [Commented] (SPARK-6948) VectorAssembler should choose dense/sparse for output based on number of zeros

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394190#comment-15394190 ] Sean Owen commented on SPARK-6948: -- I tend to agree, ran into this just recently in the exact same

[jira] [Commented] (SPARK-6948) VectorAssembler should choose dense/sparse for output based on number of zeros

2016-07-26 Thread Max Moroz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394120#comment-15394120 ] Max Moroz commented on SPARK-6948: -- Should this be a parameter for `transform` method of the

[jira] [Commented] (SPARK-16652) JVM crash from unsafe memory access for Dataset of class with List[Long]

2016-07-26 Thread Daniel Barclay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394121#comment-15394121 ] Daniel Barclay commented on SPARK-16652: Not that I can really tell either, but this does not

[jira] [Commented] (SPARK-16676) Spark jobs stay in pending

2016-07-26 Thread Joe Chong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394094#comment-15394094 ] Joe Chong commented on SPARK-16676: --- Unfortunately, that's all the logs available in the console, as

[jira] [Commented] (SPARK-16725) Migrate Guava to 16+

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394084#comment-15394084 ] Min Wei commented on SPARK-16725: - Thanks for editing the JIRA fields properly. I used the first/default

[jira] [Updated] (SPARK-16725) Migrate Guava to 16+?

2016-07-26 Thread Min Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Wei updated SPARK-16725: Summary: Migrate Guava to 16+? (was: Migrate Guava to 16+) > Migrate Guava to 16+? >

[jira] [Commented] (SPARK-16652) JVM crash from unsafe memory access for Dataset of class with List[Long]

2016-07-26 Thread Daniel Barclay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394082#comment-15394082 ] Daniel Barclay commented on SPARK-16652: > This could be reproduced at my side with Spark 1.6.1

[jira] [Created] (SPARK-16738) Queryable state for Spark State Store

2016-07-26 Thread Mark Sumner (JIRA)
Mark Sumner created SPARK-16738: --- Summary: Queryable state for Spark State Store Key: SPARK-16738 URL: https://issues.apache.org/jira/browse/SPARK-16738 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-15271) Allow force pulling executor docker images

2016-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15271. --- Resolution: Fixed Issue resolved by pull request 14348 [https://github.com/apache/spark/pull/14348]

[jira] [Assigned] (SPARK-16713) Limit codegen method size to 8KB

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16713: Assignee: (was: Apache Spark) > Limit codegen method size to 8KB >

[jira] [Commented] (SPARK-16713) Limit codegen method size to 8KB

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393863#comment-15393863 ] Apache Spark commented on SPARK-16713: -- User 'lw-lin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16713) Limit codegen method size to 8KB

2016-07-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16713: Assignee: Apache Spark > Limit codegen method size to 8KB >

[jira] [Commented] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

2016-07-26 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393770#comment-15393770 ] Steve Loughran commented on SPARK-16736: See also HIVE-14323 > remove redundant FileSystem

[jira] [Updated] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

2016-07-26 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-16736: --- Summary: remove redundant FileSystem status checks calls from Spark codebase (was: remove

[jira] [Created] (SPARK-16737) ListingFileCatalog comments about RPC calls in object store isn't correct

2016-07-26 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-16737: -- Summary: ListingFileCatalog comments about RPC calls in object store isn't correct Key: SPARK-16737 URL: https://issues.apache.org/jira/browse/SPARK-16737

[jira] [Updated] (SPARK-16737) ListingFileCatalog comments about RPC calls in object store isn't correct

2016-07-26 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-16737: --- Description: The comment text which came in with SPARK-16121 says {code} - Although

[jira] [Updated] (SPARK-16717) Dataframe (jdbc) is missing a way to link and external function to get a connection

2016-07-26 Thread Marco Colombo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Colombo updated SPARK-16717: -- Description: In JdbcRRD it was possible to use a function to get a JDBC connection. This

[jira] [Commented] (SPARK-16717) Dataframe (jdbc) is missing a way to link and external function to get a connection

2016-07-26 Thread Marco Colombo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393696#comment-15393696 ] Marco Colombo commented on SPARK-16717: --- Typo, I mean 1.6.2 > Dataframe (jdbc) is missing a way to

[jira] [Updated] (SPARK-16717) Dataframe (jdbc) is missing a way to link and external function to get a connection

2016-07-26 Thread Marco Colombo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Colombo updated SPARK-16717: -- Affects Version/s: (was: 1.3.0) 1.6.2 > Dataframe (jdbc) is missing

[jira] [Comment Edited] (SPARK-16735) Fail to create a map contains decimal type with literals having different inferred precessions and scales

2016-07-26 Thread Liang Ke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393639#comment-15393639 ] Liang Ke edited comment on SPARK-16735 at 7/26/16 12:00 PM: hi, if some one

  1   2   >