[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486506#comment-15486506 ] Tathagata Das commented on SPARK-15406: --- I tried implementing stuff as well, and I

[jira] [Created] (SPARK-17519) [MESOS] Enhance robustness when ExternalShuffleService is broken

2016-09-13 Thread Sun Rui (JIRA)
Sun Rui created SPARK-17519: --- Summary: [MESOS] Enhance robustness when ExternalShuffleService is broken Key: SPARK-17519 URL: https://issues.apache.org/jira/browse/SPARK-17519 Project: Spark Issue

[jira] [Commented] (SPARK-16750) ML GaussianMixture training failed due to feature column type mistake

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486623#comment-15486623 ] Sean Owen commented on SPARK-16750: --- You can see above that it is resolved for 2.0.1, n

[jira] [Resolved] (SPARK-2352) [MLLIB] Add Artificial Neural Network (ANN) to Spark

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2352. -- Resolution: Duplicate > [MLLIB] Add Artificial Neural Network (ANN) to Spark > -

[jira] [Commented] (SPARK-2352) [MLLIB] Add Artificial Neural Network (ANN) to Spark

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486648#comment-15486648 ] Sean Owen commented on SPARK-2352: -- There is already https://issues.apache.org/jira/brows

[jira] [Resolved] (SPARK-4288) Add Sparse Autoencoder algorithm to MLlib

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4288. -- Resolution: Duplicate > Add Sparse Autoencoder algorithm to MLlib > ---

[jira] [Resolved] (SPARK-2623) Stacked Auto Encoder (Deep Learning )

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2623. -- Resolution: Duplicate > Stacked Auto Encoder (Deep Learning ) > - >

[jira] [Updated] (SPARK-10408) Autoencoder

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10408: -- Priority: Major (was: Minor) Issue Type: Improvement (was: Umbrella) > Autoencoder > --

[jira] [Commented] (SPARK-17469) mapWithState causes block lock warning

2016-09-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486668#comment-15486668 ] Christopher Mårtensson commented on SPARK-17469: nc -lk 1234 &; ./bin/run

[jira] [Updated] (SPARK-17469) mapWithState causes block lock warning

2016-09-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Mårtensson updated SPARK-17469: --- Description: run-example with StatefulNetworkWordCount ( for example {{nc -lk

[jira] [Updated] (SPARK-17469) mapWithState causes block lock warning

2016-09-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Mårtensson updated SPARK-17469: --- Description: run-example with StatefulNetworkWordCount ( for example {{nc -lk

[jira] [Commented] (SPARK-17465) Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak

2016-09-13 Thread Xing Shi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486680#comment-15486680 ] Xing Shi commented on SPARK-17465: -- It seems that the issue has some negative side effec

[jira] [Created] (SPARK-17520) Create a more performant __eq__ for SparseMatrix

2016-09-13 Thread Giovanni Lanzani (JIRA)
Giovanni Lanzani created SPARK-17520: Summary: Create a more performant __eq__ for SparseMatrix Key: SPARK-17520 URL: https://issues.apache.org/jira/browse/SPARK-17520 Project: Spark Issu

[jira] [Updated] (SPARK-17520) Create a more performant __eq__ for SparseMatrix

2016-09-13 Thread Giovanni Lanzani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Lanzani updated SPARK-17520: - Description: Right now the SparseMatrix implementation has a slow __eq__ method (marked s

[jira] [Assigned] (SPARK-17520) Create a more performant __eq__ for SparseMatrix

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17520: Assignee: (was: Apache Spark) > Create a more performant __eq__ for SparseMatrix > ---

[jira] [Assigned] (SPARK-17520) Create a more performant __eq__ for SparseMatrix

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17520: Assignee: Apache Spark > Create a more performant __eq__ for SparseMatrix > --

[jira] [Commented] (SPARK-17520) Create a more performant __eq__ for SparseMatrix

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486723#comment-15486723 ] Apache Spark commented on SPARK-17520: -- User 'gglanzani' has created a pull request

[jira] [Commented] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486726#comment-15486726 ] Apache Spark commented on SPARK-17445: -- User 'srowen' has created a pull request for

[jira] [Assigned] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17445: Assignee: (was: Apache Spark) > Reference an ASF page as the main place to find third-

[jira] [Commented] (SPARK-17114) Adding a 'GROUP BY 1' where first column is literal results in wrong answer

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486727#comment-15486727 ] Apache Spark commented on SPARK-17114: -- User 'hvanhovell' has created a pull request

[jira] [Assigned] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17445: Assignee: Apache Spark > Reference an ASF page as the main place to find third-party packa

[jira] [Assigned] (SPARK-17114) Adding a 'GROUP BY 1' where first column is literal results in wrong answer

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17114: Assignee: (was: Apache Spark) > Adding a 'GROUP BY 1' where first column is literal re

[jira] [Assigned] (SPARK-17114) Adding a 'GROUP BY 1' where first column is literal results in wrong answer

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17114: Assignee: Apache Spark > Adding a 'GROUP BY 1' where first column is literal results in wr

[jira] [Created] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread WangJianfei (JIRA)
WangJianfei created SPARK-17521: --- Summary: Error when I use sparkContext.makeRDD(Seq()) Key: SPARK-17521 URL: https://issues.apache.org/jira/browse/SPARK-17521 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

2016-09-13 Thread Sun Rui (JIRA)
Sun Rui created SPARK-17522: --- Summary: [MESOS] More even distribution of executors on Mesos cluster Key: SPARK-17522 URL: https://issues.apache.org/jira/browse/SPARK-17522 Project: Spark Issue Typ

[jira] [Updated] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangJianfei updated SPARK-17521: Description: when i use sc.makeRDD below ``` val data3 = sc.makeRDD(Seq()) println(data3.pa

[jira] [Updated] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

2016-09-13 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sun Rui updated SPARK-17522: Description: The MesosCoarseGrainedSchedulerBackend launch executors in a round-robin way among accepted o

[jira] [Commented] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486758#comment-15486758 ] Sean Owen commented on SPARK-17445: --- I renamed the wiki to https://cwiki.apache.org/co

[jira] [Commented] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486773#comment-15486773 ] Sean Owen commented on SPARK-17521: --- Wow, I admit I didn't even know this overload exis

[jira] [Updated] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17508: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) I agree it ends up being more o

[jira] [Resolved] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17185. --- Resolution: Won't Fix I think the point is reasonable but I'm pretty sure we would not change the API

[jira] [Resolved] (SPARK-17413) spark-shell loses gnu readline support after suspend and continue

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17413. --- Resolution: Not A Problem > spark-shell loses gnu readline support after suspend and continue > -

[jira] [Resolved] (SPARK-17412) FsHistoryProviderSuite - FAILED

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17412. --- Resolution: Not A Problem > FsHistoryProviderSuite - FAILED > --- > >

[jira] [Reopened] (SPARK-17412) FsHistoryProviderSuite - FAILED

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-17412: --- > FsHistoryProviderSuite - FAILED > --- > > Key: SPARK-17412

[jira] [Commented] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486894#comment-15486894 ] WangJianfei commented on SPARK-17521: - Hi @Sean Owen I can do this fix:just deprecete

[jira] [Updated] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17521: -- Flags: (was: Important) Priority: Trivial (was: Minor) Don't do the deprecation now, that was

[jira] [Resolved] (SPARK-17515) CollectLimit.execute() should perform per-partition limits

2016-09-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17515. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > CollectLimit

[jira] [Updated] (SPARK-17517) Improve generated Code for BroadcastHashJoinExec

2016-09-13 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17517: - Description: For current `BroadcastHashJoinExec`, we generate join code for key is not unique like this:

[jira] [Created] (SPARK-17523) Cannot get Spark build info from spark-core package which built in Windows

2016-09-13 Thread Yun Tang (JIRA)
Yun Tang created SPARK-17523: Summary: Cannot get Spark build info from spark-core package which built in Windows Key: SPARK-17523 URL: https://issues.apache.org/jira/browse/SPARK-17523 Project: Spark

[jira] [Commented] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486970#comment-15486970 ] Apache Spark commented on SPARK-17521: -- User 'codlife' has created a pull request fo

[jira] [Assigned] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17521: Assignee: Apache Spark > Error when I use sparkContext.makeRDD(Seq()) > --

[jira] [Assigned] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17521: Assignee: (was: Apache Spark) > Error when I use sparkContext.makeRDD(Seq()) > ---

[jira] [Assigned] (SPARK-17523) Cannot get Spark build info from spark-core package which built in Windows

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17523: Assignee: (was: Apache Spark) > Cannot get Spark build info from spark-core package wh

[jira] [Assigned] (SPARK-17523) Cannot get Spark build info from spark-core package which built in Windows

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17523: Assignee: Apache Spark > Cannot get Spark build info from spark-core package which built i

[jira] [Commented] (SPARK-17523) Cannot get Spark build info from spark-core package which built in Windows

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486988#comment-15486988 ] Apache Spark commented on SPARK-17523: -- User 'Myasuka' has created a pull request fo

[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

2016-09-13 Thread Mayank Shete (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486990#comment-15486990 ] Mayank Shete commented on SPARK-10063: -- How can you achieve this in AWS EMR 5.0 whil

[jira] [Commented] (SPARK-17471) Add compressed method for Matrix class

2016-09-13 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487035#comment-15487035 ] Yanbo Liang commented on SPARK-17471: - [~sethah] I'm sorry that I have some emergent

[jira] [Created] (SPARK-17524) RowBasedKeyValueBatchSuite always uses 64 mb page size

2016-09-13 Thread Adam Roberts (JIRA)
Adam Roberts created SPARK-17524: Summary: RowBasedKeyValueBatchSuite always uses 64 mb page size Key: SPARK-17524 URL: https://issues.apache.org/jira/browse/SPARK-17524 Project: Spark Issue

[jira] [Commented] (SPARK-17521) Error when I use sparkContext.makeRDD(Seq())

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487089#comment-15487089 ] Sean Owen commented on SPARK-17521: --- Reaching way back to page [~pwendell] -- do you ha

[jira] [Created] (SPARK-17525) SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0

2016-09-13 Thread Sami Jaktholm (JIRA)
Sami Jaktholm created SPARK-17525: - Summary: SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0 Key: SPARK-17525 URL: https://issues.apache.

[jira] [Commented] (SPARK-17524) RowBasedKeyValueBatchSuite always uses 64 mb page size

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487093#comment-15487093 ] Apache Spark commented on SPARK-17524: -- User 'a-roberts' has created a pull request

[jira] [Assigned] (SPARK-17524) RowBasedKeyValueBatchSuite always uses 64 mb page size

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17524: Assignee: Apache Spark > RowBasedKeyValueBatchSuite always uses 64 mb page size >

[jira] [Assigned] (SPARK-17524) RowBasedKeyValueBatchSuite always uses 64 mb page size

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17524: Assignee: (was: Apache Spark) > RowBasedKeyValueBatchSuite always uses 64 mb page size

[jira] [Commented] (SPARK-17525) SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0

2016-09-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487117#comment-15487117 ] Sean Owen commented on SPARK-17525: --- Oops, yeah this was missed in https://github.com/

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487166#comment-15487166 ] Alexander Kasper commented on SPARK-17321: -- No, we're not using NM recovery. Wha

[jira] [Commented] (SPARK-16938) Cannot resolve column name after a join

2016-09-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487185#comment-15487185 ] Dongjoon Hyun commented on SPARK-16938: --- Hi, @cloud-fan . Could you review this iss

[jira] [Comment Edited] (SPARK-16938) Cannot resolve column name after a join

2016-09-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487185#comment-15487185 ] Dongjoon Hyun edited comment on SPARK-16938 at 9/13/16 1:23 PM: ---

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487203#comment-15487203 ] Thomas Graves commented on SPARK-17321: --- yes that makes sense and as I stated I thi

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2016-09-13 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487224#comment-15487224 ] Dhruve Ashar commented on SPARK-16441: -- There was a patch which was recently contrib

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487310#comment-15487310 ] Cody Koeninger commented on SPARK-15406: So we can do the easiest thing possible

[jira] [Resolved] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17142. --- Resolution: Fixed Assignee: Jiang Xingbo Fix Version/s: 2.1.0 > Compl

[jira] [Created] (SPARK-17526) Display the executor log links with the job failure message on Spark UI and Console

2016-09-13 Thread Zhan Zhang (JIRA)
Zhan Zhang created SPARK-17526: -- Summary: Display the executor log links with the job failure message on Spark UI and Console Key: SPARK-17526 URL: https://issues.apache.org/jira/browse/SPARK-17526 Proje

[jira] [Commented] (SPARK-17397) Show example of what to do when awaitTermination() throws an Exception

2016-09-13 Thread Spiro Michaylov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487691#comment-15487691 ] Spiro Michaylov commented on SPARK-17397: - In the form of a PR? Sure, I can do th

[jira] [Assigned] (SPARK-10408) Autoencoder

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10408: Assignee: Alexander Ulanov (was: Apache Spark) > Autoencoder > --- > >

[jira] [Commented] (SPARK-10408) Autoencoder

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487696#comment-15487696 ] Apache Spark commented on SPARK-10408: -- User 'avulanov' has created a pull request f

[jira] [Assigned] (SPARK-10408) Autoencoder

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10408: Assignee: Apache Spark (was: Alexander Ulanov) > Autoencoder > --- > >

[jira] [Comment Edited] (SPARK-17397) Show example of what to do when awaitTermination() throws an Exception

2016-09-13 Thread Spiro Michaylov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487691#comment-15487691 ] Spiro Michaylov edited comment on SPARK-17397 at 9/13/16 4:37 PM: -

[jira] [Assigned] (SPARK-17526) Display the executor log links with the job failure message on Spark UI and Console

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17526: Assignee: Apache Spark > Display the executor log links with the job failure message on Sp

[jira] [Commented] (SPARK-17526) Display the executor log links with the job failure message on Spark UI and Console

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487705#comment-15487705 ] Apache Spark commented on SPARK-17526: -- User 'zhzhan' has created a pull request for

[jira] [Assigned] (SPARK-17526) Display the executor log links with the job failure message on Spark UI and Console

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17526: Assignee: (was: Apache Spark) > Display the executor log links with the job failure me

[jira] [Commented] (SPARK-17252) Performing arithmetic in VALUES can lead to ClassCastException / MatchErrors during query parsing

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487749#comment-15487749 ] Apache Spark commented on SPARK-17252: -- User 'sjakthol' has created a pull request f

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Frederick Reiss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487780#comment-15487780 ] Frederick Reiss commented on SPARK-15406: - +1 for taking the simple route in the

[jira] [Assigned] (SPARK-17525) SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17525: Assignee: Apache Spark > SparkContext.clearFiles() still present in the PySpark bindings t

[jira] [Assigned] (SPARK-17525) SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17525: Assignee: (was: Apache Spark) > SparkContext.clearFiles() still present in the PySpark

[jira] [Commented] (SPARK-17525) SparkContext.clearFiles() still present in the PySpark bindings though the underlying Scala method was removed in Spark 2.0

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487789#comment-15487789 ] Apache Spark commented on SPARK-17525: -- User 'sjakthol' has created a pull request f

[jira] [Created] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-13 Thread Gaurav Shah (JIRA)
Gaurav Shah created SPARK-17527: --- Summary: mergeSchema with `_OPTIONAL_` metadata fails Key: SPARK-17527 URL: https://issues.apache.org/jira/browse/SPARK-17527 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17528) MutableProjection should not cache content from the input row

2016-09-13 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-17528: --- Summary: MutableProjection should not cache content from the input row Key: SPARK-17528 URL: https://issues.apache.org/jira/browse/SPARK-17528 Project: Spark

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487875#comment-15487875 ] Michael Armbrust commented on SPARK-15406: -- Hey Cody, thanks for the input and f

[jira] [Commented] (SPARK-17528) MutableProjection should not cache content from the input row

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487907#comment-15487907 ] Apache Spark commented on SPARK-17528: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-17528) MutableProjection should not cache content from the input row

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17528: Assignee: Apache Spark (was: Wenchen Fan) > MutableProjection should not cache content fr

[jira] [Assigned] (SPARK-17528) MutableProjection should not cache content from the input row

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17528: Assignee: Wenchen Fan (was: Apache Spark) > MutableProjection should not cache content fr

[jira] [Updated] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-15406: -- Description: This is the parent JIRA to track all the work for the building a Kafka source for

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487930#comment-15487930 ] Cody Koeninger commented on SPARK-15406: Specific examples: Kafka has a type for

[jira] [Commented] (SPARK-17450) spark sql rownumber OOM

2016-09-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488005#comment-15488005 ] Herman van Hovell commented on SPARK-17450: --- https://github.com/apache/spark/pu

[jira] [Commented] (SPARK-10816) API design: window and session specification

2016-09-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488035#comment-15488035 ] Maciej Bryński commented on SPARK-10816: Hi, Any updates on Session Window ? > A

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488079#comment-15488079 ] Tathagata Das commented on SPARK-15406: --- Here are some thoughts. - The key-value

[jira] [Created] (SPARK-17529) On highly skewed data, outer join merges are slow

2016-09-13 Thread David C Navas (JIRA)
David C Navas created SPARK-17529: - Summary: On highly skewed data, outer join merges are slow Key: SPARK-17529 URL: https://issues.apache.org/jira/browse/SPARK-17529 Project: Spark Issue Typ

[jira] [Created] (SPARK-17530) Add Statistics into DESCRIBE FORMATTED

2016-09-13 Thread Xiao Li (JIRA)
Xiao Li created SPARK-17530: --- Summary: Add Statistics into DESCRIBE FORMATTED Key: SPARK-17530 URL: https://issues.apache.org/jira/browse/SPARK-17530 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-17530) Add Statistics into DESCRIBE FORMATTED

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488090#comment-15488090 ] Apache Spark commented on SPARK-17530: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-17530) Add Statistics into DESCRIBE FORMATTED

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17530: Assignee: Apache Spark > Add Statistics into DESCRIBE FORMATTED >

[jira] [Assigned] (SPARK-17530) Add Statistics into DESCRIBE FORMATTED

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17530: Assignee: (was: Apache Spark) > Add Statistics into DESCRIBE FORMATTED > -

[jira] [Updated] (SPARK-17529) On highly skewed data, outer join merges are slow

2016-09-13 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-17529: - Priority: Major (was: Trivial) > On highly skewed data, outer join merges are slow > ---

[jira] [Assigned] (SPARK-17529) On highly skewed data, outer join merges are slow

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17529: Assignee: (was: Apache Spark) > On highly skewed data, outer join merges are slow > --

[jira] [Assigned] (SPARK-17529) On highly skewed data, outer join merges are slow

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17529: Assignee: Apache Spark > On highly skewed data, outer join merges are slow > -

[jira] [Commented] (SPARK-17529) On highly skewed data, outer join merges are slow

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488106#comment-15488106 ] Apache Spark commented on SPARK-17529: -- User 'davidnavas' has created a pull request

[jira] [Created] (SPARK-17531) Don't initialize Hive Listeners for the Execution Client

2016-09-13 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17531: --- Summary: Don't initialize Hive Listeners for the Execution Client Key: SPARK-17531 URL: https://issues.apache.org/jira/browse/SPARK-17531 Project: Spark Issue

[jira] [Updated] (SPARK-17531) Don't initialize Hive Listeners for the Execution Client

2016-09-13 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-17531: Description: If a user provides listeners inside the Hive Conf, the configuration for these listen

[jira] [Commented] (SPARK-17484) Race condition when cancelling a job during a cache write can lead to block fetch failures

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488273#comment-15488273 ] Apache Spark commented on SPARK-17484: -- User 'JoshRosen' has created a pull request

[jira] [Assigned] (SPARK-17484) Race condition when cancelling a job during a cache write can lead to block fetch failures

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17484: Assignee: Apache Spark (was: Josh Rosen) > Race condition when cancelling a job during a

[jira] [Assigned] (SPARK-17484) Race condition when cancelling a job during a cache write can lead to block fetch failures

2016-09-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17484: Assignee: Josh Rosen (was: Apache Spark) > Race condition when cancelling a job during a

  1   2   >