[jira] [Updated] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Selin updated SPARK-12089: --- Description: When running a large spark sql query including multiple joins I see tasks failing with

[jira] [Comment Edited] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036015#comment-15036015 ] Erik Selin edited comment on SPARK-12089 at 12/2/15 4:15 PM: - I can make that

[jira] [Comment Edited] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036015#comment-15036015 ] Erik Selin edited comment on SPARK-12089 at 12/2/15 4:07 PM: - I can make that

[jira] [Assigned] (SPARK-12096) remove the old constraint in word2vec

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12096: Assignee: Apache Spark > remove the old constraint in word2vec >

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036015#comment-15036015 ] Erik Selin commented on SPARK-12089: I can make that change if it is that easy. I'm just wondering if

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036004#comment-15036004 ] Wenchen Fan commented on SPARK-12089: - yea, I think we should be more conservative to grow the buffer

[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB

2015-12-02 Thread Christoph Pirkl (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035873#comment-15035873 ] Christoph Pirkl commented on SPARK-10969: - While this commit is useful it does not fix this

[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB

2015-12-02 Thread Brian London (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035963#comment-15035963 ] Brian London commented on SPARK-10969: -- I'm not sure why it needs to be an object as opposed to two

[jira] [Comment Edited] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB

2015-12-02 Thread Brian London (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035963#comment-15035963 ] Brian London edited comment on SPARK-10969 at 12/2/15 3:31 PM: --- I'm not

[jira] [Created] (SPARK-12098) Cross validator with multi-arm bandit search

2015-12-02 Thread Xusen Yin (JIRA)
Xusen Yin created SPARK-12098: - Summary: Cross validator with multi-arm bandit search Key: SPARK-12098 URL: https://issues.apache.org/jira/browse/SPARK-12098 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-12096) remove the old constraint in word2vec

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12096: Assignee: (was: Apache Spark) > remove the old constraint in word2vec >

[jira] [Commented] (SPARK-12096) remove the old constraint in word2vec

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035864#comment-15035864 ] Apache Spark commented on SPARK-12096: -- User 'hhbyyh' has created a pull request for this issue:

[jira] [Updated] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Selin updated SPARK-12089: --- Description: When running a large spark sql query including multiple joins I see tasks failing with

[jira] [Created] (SPARK-12097) How to do a cached, batched JDBC-lookup in Spark Streaming?

2015-12-02 Thread Christian Kurz (JIRA)
Christian Kurz created SPARK-12097: -- Summary: How to do a cached, batched JDBC-lookup in Spark Streaming? Key: SPARK-12097 URL: https://issues.apache.org/jira/browse/SPARK-12097 Project: Spark

[jira] [Commented] (SPARK-12040) Add toJson/fromJson to Vector/Vectors for PySpark

2015-12-02 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036337#comment-15036337 ] holdenk commented on SPARK-12040: - Working on this :) > Add toJson/fromJson to Vector/Vectors for

[jira] [Resolved] (SPARK-12094) Better format for query plan tree string

2015-12-02 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-12094. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 10099

[jira] [Updated] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-12089: --- Priority: Critical (was: Major) > java.lang.NegativeArraySizeException when growing BufferHolder >

[jira] [Commented] (SPARK-12097) How to do a cached, batched JDBC-lookup in Spark Streaming?

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036116#comment-15036116 ] Sean Owen commented on SPARK-12097: --- Normally I'd say we don't use JIRA for discussion (i.e. we don't

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036194#comment-15036194 ] Davies Liu commented on SPARK-12089: Is it possible that you have a record larger than 1G? I don't

[jira] [Created] (SPARK-12099) Standalone and Mesos Should use OnOutOfMemoryError handlers

2015-12-02 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-12099: Summary: Standalone and Mesos Should use OnOutOfMemoryError handlers Key: SPARK-12099 URL: https://issues.apache.org/jira/browse/SPARK-12099 Project: Spark

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036247#comment-15036247 ] Erik Selin commented on SPARK-12089: There shouldn't be a single record larger than 1G no. But I'm

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036295#comment-15036295 ] Davies Liu commented on SPARK-12089: [~tyro89] Are you build a large Array using group by? How is the

[jira] [Created] (SPARK-12100) bug in spark/python/pyspark/rdd.py portable_hash()

2015-12-02 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-12100: --- Summary: bug in spark/python/pyspark/rdd.py portable_hash() Key: SPARK-12100 URL: https://issues.apache.org/jira/browse/SPARK-12100 Project: Spark

[jira] [Assigned] (SPARK-12040) Add toJson/fromJson to Vector/Vectors for PySpark

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12040: Assignee: Apache Spark > Add toJson/fromJson to Vector/Vectors for PySpark >

[jira] [Commented] (SPARK-12040) Add toJson/fromJson to Vector/Vectors for PySpark

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036338#comment-15036338 ] Apache Spark commented on SPARK-12040: -- User 'holdenk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12040) Add toJson/fromJson to Vector/Vectors for PySpark

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12040: Assignee: (was: Apache Spark) > Add toJson/fromJson to Vector/Vectors for PySpark >

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036280#comment-15036280 ] Davies Liu commented on SPARK-12089: Could you turn on debug log, and paste the java source code of

[jira] [Assigned] (SPARK-12098) Cross validator with multi-arm bandit search

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12098: Assignee: (was: Apache Spark) > Cross validator with multi-arm bandit search >

[jira] [Commented] (SPARK-12098) Cross validator with multi-arm bandit search

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036231#comment-15036231 ] Apache Spark commented on SPARK-12098: -- User 'yinxusen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12098) Cross validator with multi-arm bandit search

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12098: Assignee: Apache Spark > Cross validator with multi-arm bandit search >

[jira] [Commented] (SPARK-11801) Notify driver when OOM is thrown before executor JVM is killed

2015-12-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036302#comment-15036302 ] Imran Rashid commented on SPARK-11801: -- to summarize, it seems we agree that: 1) we want to keep

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Erik Selin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036327#comment-15036327 ] Erik Selin commented on SPARK-12089: It's a bunch of table joins followed by a group by on multiple

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2015-12-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036341#comment-15036341 ] Thomas Graves commented on SPARK-1239: -- I have another user hitting this also. The above mentions

[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB

2015-12-02 Thread Christoph Pirkl (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036418#comment-15036418 ] Christoph Pirkl commented on SPARK-10969: - This issue is about adding additional (optional)

[jira] [Updated] (SPARK-11219) Make Parameter Description Format Consistent in PySpark.MLlib

2015-12-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-11219: - Description: There are several different formats for describing params in PySpark.MLlib, making

[jira] [Commented] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036487#comment-15036487 ] Sean Owen commented on SPARK-12101: --- To cut down the noise, if this is logically the same issue as one

[jira] [Updated] (SPARK-10873) can't sort columns on history page

2015-12-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-10873: -- Assignee: Zhuo Liu > can't sort columns on history page > -- >

[jira] [Commented] (SPARK-11155) Stage summary json should include stage duration

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036375#comment-15036375 ] Apache Spark commented on SPARK-11155: -- User 'keypointt' has created a pull request for this issue:

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Description: (Note: yes, there is a Direct API that may be better, but it's not the easiest thing

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

2015-12-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036395#comment-15036395 ] Thomas Graves commented on SPARK-11701: --- Also seems related to

[jira] [Updated] (SPARK-11155) Stage summary json should include stage duration

2015-12-02 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren updated SPARK-11155: Attachment: Screen Shot 2015-12-02.png sorry for being so slow... my first code change and it took me

[jira] [Assigned] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12101: Assignee: Apache Spark (was: Shixiong Zhu) > Fix thread pools that cannot cache tasks in

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Fix Version/s: (was: 1.0.1) 1.4.2 > KafkaUtils createStream with multiple

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Affects Version/s: 1.1.0 1.2.0 1.3.0

[jira] [Created] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-12101: Summary: Fix thread pools that cannot cache tasks in Worker and AppClient Key: SPARK-12101 URL: https://issues.apache.org/jira/browse/SPARK-12101 Project: Spark

[jira] [Created] (SPARK-12102) In check analysis, data type check always

2015-12-02 Thread Yin Huai (JIRA)
Yin Huai created SPARK-12102: Summary: In check analysis, data type check always Key: SPARK-12102 URL: https://issues.apache.org/jira/browse/SPARK-12102 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-12102) Cast a non-nullable struct field to a nullable field during analysis

2015-12-02 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12102: - Summary: Cast a non-nullable struct field to a nullable field during analysis (was: In check analysis,

[jira] [Updated] (SPARK-12000) `sbt publishLocal` hits a Scala compiler bug caused by `Since` annotation

2015-12-02 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-12000: - Target Version/s: 1.7.0 (was: 1.6.0) > `sbt publishLocal` hits a Scala compiler bug

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

2015-12-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036366#comment-15036366 ] Thomas Graves commented on SPARK-11701: --- this looks like a dup of SPARK-9038 > YARN - dynamic

[jira] [Updated] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12101: -- Priority: Minor (was: Major) > Fix thread pools that cannot cache tasks in Worker and AppClient >

[jira] [Assigned] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12101: Assignee: Shixiong Zhu (was: Apache Spark) > Fix thread pools that cannot cache tasks in

[jira] [Commented] (SPARK-12101) Fix thread pools that cannot cache tasks in Worker and AppClient

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036516#comment-15036516 ] Apache Spark commented on SPARK-12101: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Description: (Note: yes, there is a Direct API that may be better, but it's not the easiest thing

[jira] [Commented] (SPARK-11219) Make Parameter Description Format Consistent in PySpark.MLlib

2015-12-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036470#comment-15036470 ] Bryan Cutler commented on SPARK-11219: -- I added an assessment of the current state of param

[jira] [Created] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
Dan Dutrow created SPARK-12103: -- Summary: KafkaUtils createStream with multiple topics -- does not work as expected Key: SPARK-12103 URL: https://issues.apache.org/jira/browse/SPARK-12103 Project: Spark

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Description: (Note: yes, there is a Direct API that may be better, but it's not the easiest thing

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Target Version/s: 1.4.2, 1.6.1 (was: 1.0.1) > KafkaUtils createStream with multiple topics -- does

[jira] [Commented] (SPARK-12088) check connection.isClose before connection.getAutoCommit in JDBCRDD.close

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035455#comment-15035455 ] Apache Spark commented on SPARK-12088: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12093) Fix the error of comment in DDLParser

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12093: Assignee: Apache Spark > Fix the error of comment in DDLParser >

[jira] [Commented] (SPARK-12093) Fix the error of comment in DDLParser

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035462#comment-15035462 ] Apache Spark commented on SPARK-12093: -- User 'watermen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12093) Fix the error of comment in DDLParser

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12093: Assignee: (was: Apache Spark) > Fix the error of comment in DDLParser >

[jira] [Commented] (SPARK-11964) Create user guide section explaining export/import

2015-12-02 Thread Bill Chambers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035480#comment-15035480 ] Bill Chambers commented on SPARK-11964: --- quick question, am I to assume that all pieces mentioned

[jira] [Commented] (SPARK-12048) JDBCRDD calls close() twice - SQLite then throws an exception

2015-12-02 Thread R. H. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035486#comment-15035486 ] R. H. commented on SPARK-12048: --- I have added the missing line, built a distribution, and tested it

[jira] [Commented] (SPARK-12067) Fix usage of isnan, isnull, isnotnull of Column and DataFrame

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035496#comment-15035496 ] Apache Spark commented on SPARK-12067: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Updated] (SPARK-12067) Fix usage of isnan, isnull, isnotnull of Column

2015-12-02 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12067: Summary: Fix usage of isnan, isnull, isnotnull of Column (was: Fix usage of isnan, isnull,

[jira] [Updated] (SPARK-12067) Fix usage of isnan, isnull, isnotnull of Column and DataFrame

2015-12-02 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12067: Description: * -SPARK-11947 has deprecated DataFrame.isNaN, DataFrame.isNull and replaced by

[jira] [Assigned] (SPARK-12088) check connection.isClose before connection.getAutoCommit in JDBCRDD.close

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12088: Assignee: Apache Spark > check connection.isClose before connection.getAutoCommit in

[jira] [Assigned] (SPARK-12088) check connection.isClose before connection.getAutoCommit in JDBCRDD.close

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12088: Assignee: (was: Apache Spark) > check connection.isClose before

[jira] [Created] (SPARK-12093) Fix the error of comment in DDLParser

2015-12-02 Thread Yadong Qi (JIRA)
Yadong Qi created SPARK-12093: - Summary: Fix the error of comment in DDLParser Key: SPARK-12093 URL: https://issues.apache.org/jira/browse/SPARK-12093 Project: Spark Issue Type: Documentation

[jira] [Comment Edited] (SPARK-11964) Create user guide section explaining export/import

2015-12-02 Thread Bill Chambers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035480#comment-15035480 ] Bill Chambers edited comment on SPARK-11964 at 12/2/15 8:52 AM: quick

[jira] [Created] (SPARK-12094) Better format for query plan tree string

2015-12-02 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-12094: -- Summary: Better format for query plan tree string Key: SPARK-12094 URL: https://issues.apache.org/jira/browse/SPARK-12094 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-12048) JDBCRDD calls close() twice - SQLite then throws an exception

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035521#comment-15035521 ] Sean Owen commented on SPARK-12048: --- You need to read

[jira] [Updated] (SPARK-12067) Fix usage of isnan, isnull, isnotnull of Column and DataFrame

2015-12-02 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12067: Description: -* SPARK-11947 has deprecated DataFrame.isNaN, DataFrame.isNull and replaced by

[jira] [Updated] (SPARK-12067) Fix usage of isnan, isnull, isnotnull of Column and DataFrame

2015-12-02 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12067: Description: * -SPARK-11947 has deprecated DataFrame.isNaN, DataFrame.isNull and replaced by

[jira] [Updated] (SPARK-12093) Fix the error of comment in DDLParser

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12093: -- Priority: Trivial (was: Major) Component/s: Documentation [~waterman] This probably isn't

[jira] [Updated] (SPARK-12094) Better format for query plan tree string

2015-12-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12094: --- Description: When examine plans of complex queries with multiple joins, a pain point of mine is

[jira] [Updated] (SPARK-12080) Kryo - Support multiple user registrators

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12080: -- Affects Version/s: (was: 1.6.1) 1.5.2 > Kryo - Support multiple user

[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown

2015-12-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036613#comment-15036613 ] Marcelo Vanzin commented on SPARK-10911: Ok, that makes sense. But since the YARN bug is the

[jira] [Created] (SPARK-12104) collect() does not handle multiple columns with same name

2015-12-02 Thread Hossein Falaki (JIRA)
Hossein Falaki created SPARK-12104: -- Summary: collect() does not handle multiple columns with same name Key: SPARK-12104 URL: https://issues.apache.org/jira/browse/SPARK-12104 Project: Spark

[jira] [Commented] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036656#comment-15036656 ] Apache Spark commented on SPARK-12106: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12106: Assignee: (was: Apache Spark) > Flaky Test: BatchedWriteAheadLog - name log with

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12103: -- Affects Version/s: (was: 1.4.0) (was: 1.3.0)

[jira] [Assigned] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

2015-12-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12106: Assignee: Apache Spark > Flaky Test: BatchedWriteAheadLog - name log with aggregated

[jira] [Updated] (SPARK-12107) Update spark-ec2 versions

2015-12-02 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-12107: - Target Version/s: 1.6.0 > Update spark-ec2 versions > - > >

[jira] [Commented] (SPARK-12085) The join condition hidden in DNF can't be pushed down to join operator

2015-12-02 Thread Min Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036685#comment-15036685 ] Min Qiu commented on SPARK-12085: - looks like the BooleanSimplification rule in Spark 1.5 provides a

[jira] [Commented] (SPARK-8517) Improve the organization and style of MLlib's user guide

2015-12-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036710#comment-15036710 ] Xiangrui Meng commented on SPARK-8517: -- * I'm not sure whether the mathematical formulation is

[jira] [Commented] (SPARK-12082) NettyBlockTransferSecuritySuite "security mismatch auth off on client" test is flaky

2015-12-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036738#comment-15036738 ] Marcelo Vanzin commented on SPARK-12082: If the build machines run on VMs, run multiple jobs, or

[jira] [Commented] (SPARK-12082) NettyBlockTransferSecuritySuite "security mismatch auth off on client" test is flaky

2015-12-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036743#comment-15036743 ] Josh Rosen commented on SPARK-12082: For now, I'm going to try bumping the timeout to something

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036807#comment-15036807 ] Davies Liu commented on SPARK-12089: This query will not generate huge record, each record should be

[jira] [Closed] (SPARK-11992) Severl numbers in my spark shell (pyspark)

2015-12-02 Thread Alberto Bonsanto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alberto Bonsanto closed SPARK-11992. > Severl numbers in my spark shell (pyspark) > -- > >

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow updated SPARK-12103: --- Description: (Note: yes, there is a Direct API that may be better, but it's not the easiest thing

[jira] [Commented] (SPARK-8517) Improve the organization and style of MLlib's user guide

2015-12-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036690#comment-15036690 ] Xiangrui Meng commented on SPARK-8517: -- * Agree that the focus of spark.ml should not only be

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-02 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036741#comment-15036741 ] Dan Dutrow commented on SPARK-12103: One possible way around this problem would be to stick the topic

[jira] [Commented] (SPARK-12104) collect() does not handle multiple columns with same name

2015-12-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036791#comment-15036791 ] Shivaram Venkataraman commented on SPARK-12104: --- Any ideas what caused this ? > collect()

[jira] [Created] (SPARK-12110) spark-1.5.1-bin-hadoop2.6; pyspark.ml.feature Exception: ("You must build Spark with Hive

2015-12-02 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-12110: --- Summary: spark-1.5.1-bin-hadoop2.6; pyspark.ml.feature Exception: ("You must build Spark with Hive Key: SPARK-12110 URL:

[jira] [Commented] (SPARK-11255) R Test build should run on R 3.1.1

2015-12-02 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036809#comment-15036809 ] shane knapp commented on SPARK-11255: - ok, [~shivaram] and i got together and whipped up a test build

[jira] [Created] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

2015-12-02 Thread Dean Wampler (JIRA)
Dean Wampler created SPARK-12105: Summary: Add a DataFrame.show() with argument for output PrintStream Key: SPARK-12105 URL: https://issues.apache.org/jira/browse/SPARK-12105 Project: Spark

[jira] [Commented] (SPARK-12066) spark sql throw java.lang.ArrayIndexOutOfBoundsException when use table.* with join

2015-12-02 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036683#comment-15036683 ] Michael Armbrust commented on SPARK-12066: -- Can you reproduce this on 1.6-rc1? > spark sql

[jira] [Updated] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

2015-12-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-12106: --- Labels: flaky-test (was: ) > Flaky Test: BatchedWriteAheadLog - name log with aggregated entries

[jira] [Commented] (SPARK-8517) Improve the organization and style of MLlib's user guide

2015-12-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036718#comment-15036718 ] Xiangrui Meng commented on SPARK-8517: -- [~timhunter] I agree with most of your points. I'd recommend

  1   2   3   >