[jira] [Updated] (SPARK-17441) Issue Exceptions when ALTER TABLE RENAME PARTITION tries to alter a data source table

2016-09-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-17441: Assignee: Xiao Li > Issue Exceptions when ALTER TABLE RENAME PARTITION tries to alter a data > sou

[jira] [Resolved] (SPARK-17440) Issue Exception when ALTER TABLE commands try to alter a VIEW

2016-09-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17440. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15004 [https://githu

[jira] [Resolved] (SPARK-17441) Issue Exceptions when ALTER TABLE RENAME PARTITION tries to alter a data source table

2016-09-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17441. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15004 [https://githu

[jira] [Updated] (SPARK-17440) Issue Exception when ALTER TABLE commands try to alter a VIEW

2016-09-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-17440: Assignee: Xiao Li > Issue Exception when ALTER TABLE commands try to alter a VIEW > ---

[jira] [Commented] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492494#comment-15492494 ] Apache Spark commented on SPARK-17543: -- User 'jagadeesanas2' has created a pull requ

[jira] [Assigned] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17543: Assignee: (was: Apache Spark) > Missing log4j config file for tests in common/network-

[jira] [Assigned] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17543: Assignee: Apache Spark > Missing log4j config file for tests in common/network-shuffle > -

[jira] [Commented] (SPARK-17537) Improve performance for reading parquet schema

2016-09-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492475#comment-15492475 ] Hyukjin Kwon commented on SPARK-17537: -- This should be a duplicate of SPARK-17071.

[jira] [Commented] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-14 Thread Jagadeesan A S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492406#comment-15492406 ] Jagadeesan A S commented on SPARK-17543: Started working on this. > Missing log4

[jira] [Assigned] (SPARK-17551) support null ordering for DataFrame API

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17551: Assignee: Apache Spark > support null ordering for DataFrame API > ---

[jira] [Commented] (SPARK-17551) support null ordering for DataFrame API

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492342#comment-15492342 ] Apache Spark commented on SPARK-17551: -- User 'xwu0226' has created a pull request fo

[jira] [Assigned] (SPARK-17551) support null ordering for DataFrame API

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17551: Assignee: (was: Apache Spark) > support null ordering for DataFrame API >

[jira] [Created] (SPARK-17551) support null ordering for DataFrame API

2016-09-14 Thread Xin Wu (JIRA)
Xin Wu created SPARK-17551: -- Summary: support null ordering for DataFrame API Key: SPARK-17551 URL: https://issues.apache.org/jira/browse/SPARK-17551 Project: Spark Issue Type: Improvement

[jira] [Issue Comment Deleted] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangJianfei updated SPARK-17535: Comment: was deleted (was: First: this logic and code is simple too. Second:After JDK1.5, The volat

[jira] [Commented] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492084#comment-15492084 ] WangJianfei commented on SPARK-17535: - First: this logic and code is simple too. Seco

[jira] [Comment Edited] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492046#comment-15492046 ] WangJianfei edited comment on SPARK-17535 at 9/15/16 2:10 AM: -

[jira] [Issue Comment Deleted] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangJianfei updated SPARK-17535: Comment: was deleted (was: After JDK1.5, The volatile can avoid the situation you say.We can't get

[jira] [Commented] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492046#comment-15492046 ] WangJianfei commented on SPARK-17535: - After JDK1.5, The volatile can avoid the situa

[jira] [Commented] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492047#comment-15492047 ] WangJianfei commented on SPARK-17535: - After JDK1.5, The volatile can avoid the situa

[jira] [Commented] (SPARK-15573) Backwards-compatible persistence for spark.ml

2016-09-14 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491974#comment-15491974 ] yuhao yang commented on SPARK-15573: This sounds feasible. Two primary work items as

[jira] [Commented] (SPARK-17544) Timeout waiting for connection from pool, DataFrame Reader's not closing S3 connections?

2016-09-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491938#comment-15491938 ] Josh Rosen commented on SPARK-17544: I found a similar issue from the {{spark-avro}}

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-14 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491841#comment-15491841 ] Bryan Cutler commented on SPARK-17508: -- To respond to [~srowen] question, I think it

[jira] [Updated] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17549: --- Attachment: spark-2.0.patch Attaching a Spark 2 patch that silences the error (looks like a J

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491819#comment-15491819 ] Michael Armbrust commented on SPARK-16407: -- I think it is likely that we will wa

[jira] [Updated] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17549: --- Attachment: spark-1.6-2.patch Just noticed there's already a more accurate count of the batch

[jira] [Created] (SPARK-17550) DataFrameWriter.partitionBy() should throw exception if column is not present in Dataframe

2016-09-14 Thread Aniket Kulkarni (JIRA)
Aniket Kulkarni created SPARK-17550: --- Summary: DataFrameWriter.partitionBy() should throw exception if column is not present in Dataframe Key: SPARK-17550 URL: https://issues.apache.org/jira/browse/SPARK-17550

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491787#comment-15491787 ] holdenk commented on SPARK-16407: - That's part of why I decided to just use the ForeachRD

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491784#comment-15491784 ] holdenk commented on SPARK-16407: - It's true it doesn't work in SQL - but I don't think t

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491779#comment-15491779 ] Reynold Xin commented on SPARK-16407: - Actually I spoke too soon. I only read the cod

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491772#comment-15491772 ] holdenk commented on SPARK-16407: - Right the simplest example where you need to use the t

[jira] [Assigned] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16439: Assignee: Apache Spark (was: Maciej Bryński) > Incorrect information in SQL Query details

[jira] [Assigned] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16439: Assignee: Maciej Bryński (was: Apache Spark) > Incorrect information in SQL Query details

[jira] [Commented] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491766#comment-15491766 ] Apache Spark commented on SPARK-16439: -- User 'davies' has created a pull request for

[jira] [Updated] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17549: --- Attachment: spark-1.6.patch example_1.6_pre_patch.png example_

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491752#comment-15491752 ] Reynold Xin commented on SPARK-16407: - This doesn't work in SQL, Python, etc. I like

[jira] [Reopened] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reopened SPARK-16439: We could bring the seperator back for better readability. > Incorrect information in SQL Query detail

[jira] [Created] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-14 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-17549: -- Summary: InMemoryRelation doesn't scale to large tables Key: SPARK-17549 URL: https://issues.apache.org/jira/browse/SPARK-17549 Project: Spark Issue Type

[jira] [Commented] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491744#comment-15491744 ] Davies Liu commented on SPARK-16439: The separator was added on purpose, otherwise it

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491739#comment-15491739 ] Shixiong Zhu commented on SPARK-16407: -- Right now we don't want to add such typed AP

[jira] [Assigned] (SPARK-17548) Word2VecModel.findSynonyms can spuriously reject the best match when invoked with a vector

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17548: Assignee: (was: Apache Spark) > Word2VecModel.findSynonyms can spuriously reject the b

[jira] [Commented] (SPARK-17548) Word2VecModel.findSynonyms can spuriously reject the best match when invoked with a vector

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491626#comment-15491626 ] Apache Spark commented on SPARK-17548: -- User 'willb' has created a pull request for

[jira] [Assigned] (SPARK-17548) Word2VecModel.findSynonyms can spuriously reject the best match when invoked with a vector

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17548: Assignee: Apache Spark > Word2VecModel.findSynonyms can spuriously reject the best match w

[jira] [Created] (SPARK-17548) Word2VecModel.findSynonyms can spuriously reject the best match when invoked with a vector

2016-09-14 Thread William Benton (JIRA)
William Benton created SPARK-17548: -- Summary: Word2VecModel.findSynonyms can spuriously reject the best match when invoked with a vector Key: SPARK-17548 URL: https://issues.apache.org/jira/browse/SPARK-17548

[jira] [Assigned] (SPARK-17547) Temporary shuffle data files may be leaked following exception in write

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17547: Assignee: Josh Rosen (was: Apache Spark) > Temporary shuffle data files may be leaked fol

[jira] [Commented] (SPARK-17547) Temporary shuffle data files may be leaked following exception in write

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491548#comment-15491548 ] Apache Spark commented on SPARK-17547: -- User 'JoshRosen' has created a pull request

[jira] [Assigned] (SPARK-17547) Temporary shuffle data files may be leaked following exception in write

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17547: Assignee: Apache Spark (was: Josh Rosen) > Temporary shuffle data files may be leaked fol

[jira] [Created] (SPARK-17547) Temporary shuffle data files may be leaked following exception in write

2016-09-14 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-17547: -- Summary: Temporary shuffle data files may be leaked following exception in write Key: SPARK-17547 URL: https://issues.apache.org/jira/browse/SPARK-17547 Project: Spark

[jira] [Commented] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491518#comment-15491518 ] Apache Spark commented on SPARK-17100: -- User 'davies' has created a pull request for

[jira] [Assigned] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17100: Assignee: Davies Liu (was: Apache Spark) > pyspark filter on a udf column after join give

[jira] [Assigned] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17100: Assignee: Apache Spark (was: Davies Liu) > pyspark filter on a udf column after join give

[jira] [Commented] (SPARK-17346) Kafka 0.10 support in Structured Streaming

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491487#comment-15491487 ] Apache Spark commented on SPARK-17346: -- User 'zsxwing' has created a pull request fo

[jira] [Assigned] (SPARK-17346) Kafka 0.10 support in Structured Streaming

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17346: Assignee: Apache Spark > Kafka 0.10 support in Structured Streaming >

[jira] [Assigned] (SPARK-17346) Kafka 0.10 support in Structured Streaming

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17346: Assignee: (was: Apache Spark) > Kafka 0.10 support in Structured Streaming > -

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Jeff Nadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491460#comment-15491460 ] Jeff Nadler commented on SPARK-17510: - That would be incredible, thank you very much

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491455#comment-15491455 ] Cody Koeninger commented on SPARK-17510: Ok, next time I get some free hacking ti

[jira] [Assigned] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-17100: -- Assignee: Davies Liu > pyspark filter on a udf column after join gives > java.lang.Unsupporte

[jira] [Updated] (SPARK-17465) Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak

2016-09-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17465: --- Fix Version/s: 2.1.0 2.0.1 > Inappropriate memory management in `org.apache.spark.

[jira] [Commented] (SPARK-17544) Timeout waiting for connection from pool, DataFrame Reader's not closing S3 connections?

2016-09-14 Thread Brady Auen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491445#comment-15491445 ] Brady Auen commented on SPARK-17544: http://stackoverflow.com/questions/39498492/spar

[jira] [Commented] (SPARK-17114) Adding a 'GROUP BY 1' where first column is literal results in wrong answer

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491437#comment-15491437 ] Apache Spark commented on SPARK-17114: -- User 'hvanhovell' has created a pull request

[jira] [Updated] (SPARK-17465) Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak

2016-09-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17465: --- Assignee: Xing Shi > Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may >

[jira] [Resolved] (SPARK-17465) Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak

2016-09-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17465. Resolution: Fixed Fix Version/s: 1.6.3 Issue resolved by pull request 15022 [https://github.

[jira] [Created] (SPARK-17546) start-* scripts should use hostname --fqdn

2016-09-14 Thread Kevin Burton (JIRA)
Kevin Burton created SPARK-17546: Summary: start-* scripts should use hostname --fqdn Key: SPARK-17546 URL: https://issues.apache.org/jira/browse/SPARK-17546 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Jeff Nadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491412#comment-15491412 ] Jeff Nadler commented on SPARK-17510: - Well... both streams use updateStateByKey.

[jira] [Created] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date with colon in offset

2016-09-14 Thread Nathan Beyer (JIRA)
Nathan Beyer created SPARK-17545: Summary: Spark SQL Catalyst doesn't handle ISO 8601 date with colon in offset Key: SPARK-17545 URL: https://issues.apache.org/jira/browse/SPARK-17545 Project: Spark

[jira] [Resolved] (SPARK-17472) Better error message for serialization failures of large objects in Python

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17472. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15026 [https://github.

[jira] [Commented] (SPARK-17544) Timeout waiting for connection from pool, DataFrame Reader's not closing S3 connections?

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491397#comment-15491397 ] Davies Liu commented on SPARK-17544: Could you post some code to reproduce the issue?

[jira] [Resolved] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17463. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull reque

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491389#comment-15491389 ] Cody Koeninger commented on SPARK-17510: Just for clarity's sake, compute time is

[jira] [Resolved] (SPARK-17511) Dynamic allocation race condition: Containers getting marked failed while releasing

2016-09-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17511. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-16424) Add support for Structured Streaming to the ML Pipeline API

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491250#comment-15491250 ] holdenk commented on SPARK-16424: - Just an update - we have a really early proof of conce

[jira] [Created] (SPARK-17544) Timeout waiting for connection from pool, DataFrame Reader's not closing S3 connections?

2016-09-14 Thread Brady Auen (JIRA)
Brady Auen created SPARK-17544: -- Summary: Timeout waiting for connection from pool, DataFrame Reader's not closing S3 connections? Key: SPARK-17544 URL: https://issues.apache.org/jira/browse/SPARK-17544

[jira] [Resolved] (SPARK-10747) add support for NULLS FIRST|LAST in ORDER BY clause

2016-09-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-10747. --- Resolution: Fixed Assignee: Xin Wu Fix Version/s: 2.1.0 > add support

[jira] [Comment Edited] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-14 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491186#comment-15491186 ] Evan Zamir edited comment on SPARK-17508 at 9/14/16 6:53 PM: -

[jira] [Comment Edited] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-14 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491186#comment-15491186 ] Evan Zamir edited comment on SPARK-17508 at 9/14/16 6:52 PM: -

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-14 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491186#comment-15491186 ] Evan Zamir commented on SPARK-17508: Honestly, if the documentation was just more exp

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-14 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491163#comment-15491163 ] Bryan Cutler commented on SPARK-17508: -- I had a similar discussion in this PR https

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Jeff Nadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491165#comment-15491165 ] Jeff Nadler commented on SPARK-17510: - Yes you're right - it's partly about differin

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2016-09-14 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491107#comment-15491107 ] Maciej Bryński commented on SPARK-16534: [~rxin] Could you explain your decision

[jira] [Comment Edited] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491087#comment-15491087 ] Cody Koeninger edited comment on SPARK-17510 at 9/14/16 6:12 PM: --

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491087#comment-15491087 ] Cody Koeninger commented on SPARK-17510: I use direct stream for multiple topic j

[jira] [Updated] (SPARK-17542) Compiler warning in UnsafeInMemorySorter class

2016-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17542: -- Issue Type: Improvement (was: Bug) There are unfortunately a number of warnings, and I don't think we

[jira] [Commented] (SPARK-17317) Add package vignette to SparkR

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490995#comment-15490995 ] Apache Spark commented on SPARK-17317: -- User 'junyangq' has created a pull request f

[jira] [Resolved] (SPARK-17514) df.take(1) and df.limit(1).collect() perform differently in Python

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17514. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull reque

[jira] [Created] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-14 Thread Frederick Reiss (JIRA)
Frederick Reiss created SPARK-17543: --- Summary: Missing log4j config file for tests in common/network-shuffle Key: SPARK-17543 URL: https://issues.apache.org/jira/browse/SPARK-17543 Project: Spark

[jira] [Created] (SPARK-17542) Compiler warning in UnsafeInMemorySorter class

2016-09-14 Thread Frederick Reiss (JIRA)
Frederick Reiss created SPARK-17542: --- Summary: Compiler warning in UnsafeInMemorySorter class Key: SPARK-17542 URL: https://issues.apache.org/jira/browse/SPARK-17542 Project: Spark Issue Ty

[jira] [Commented] (SPARK-17541) fix some DDL bugs about table management when same-name temp view exists

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490922#comment-15490922 ] Apache Spark commented on SPARK-17541: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-17541) fix some DDL bugs about table management when same-name temp view exists

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17541: Assignee: Wenchen Fan (was: Apache Spark) > fix some DDL bugs about table management when

[jira] [Assigned] (SPARK-17541) fix some DDL bugs about table management when same-name temp view exists

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17541: Assignee: Apache Spark (was: Wenchen Fan) > fix some DDL bugs about table management when

[jira] [Created] (SPARK-17541) fix some DDL bugs about table management when same-name temp view exists

2016-09-14 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-17541: --- Summary: fix some DDL bugs about table management when same-name temp view exists Key: SPARK-17541 URL: https://issues.apache.org/jira/browse/SPARK-17541 Project: Spark

[jira] [Assigned] (SPARK-17536) Minor performance improvement to JDBC batch inserts

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17536: Assignee: (was: Apache Spark) > Minor performance improvement to JDBC batch inserts >

[jira] [Commented] (SPARK-17536) Minor performance improvement to JDBC batch inserts

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490849#comment-15490849 ] Apache Spark commented on SPARK-17536: -- User 'blue666man' has created a pull request

[jira] [Assigned] (SPARK-17536) Minor performance improvement to JDBC batch inserts

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17536: Assignee: Apache Spark > Minor performance improvement to JDBC batch inserts > ---

[jira] [Commented] (SPARK-15835) The read path of json doesn't support write path when schema contains Options

2016-09-14 Thread Chris Horn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490836#comment-15490836 ] Chris Horn commented on SPARK-15835: You can work around this issue by providing the

[jira] [Assigned] (SPARK-17540) SparkR array serde cannot work correctly when array length == 0

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17540: Assignee: Apache Spark > SparkR array serde cannot work correctly when array length == 0 >

[jira] [Assigned] (SPARK-17540) SparkR array serde cannot work correctly when array length == 0

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17540: Assignee: (was: Apache Spark) > SparkR array serde cannot work correctly when array le

[jira] [Commented] (SPARK-17540) SparkR array serde cannot work correctly when array length == 0

2016-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490790#comment-15490790 ] Apache Spark commented on SPARK-17540: -- User 'WeichenXu123' has created a pull reque

[jira] [Created] (SPARK-17540) SparkR array serde cannot work correctly when array length == 0

2016-09-14 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-17540: -- Summary: SparkR array serde cannot work correctly when array length == 0 Key: SPARK-17540 URL: https://issues.apache.org/jira/browse/SPARK-17540 Project: Spark

[jira] [Commented] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

2016-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490737#comment-15490737 ] Sean Owen commented on SPARK-17535: --- I think this happens to work out because the conte

[jira] [Updated] (SPARK-17538) sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0

2016-09-14 Thread Srinivas Rishindra Pothireddi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-17538: -- Description: I have a production job in spark 1.6.2 that regist

[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-14 Thread Jeff Nadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490746#comment-15490746 ] Jeff Nadler commented on SPARK-17510: - I filed SPARK-17539 for the backpressure bug.

  1   2   >