[jira] [Commented] (SPARK-3888) Limit the memory used by python worker

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173454#comment-14173454 ] Apache Spark commented on SPARK-3888: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173489#comment-14173489 ] Mridul Muralidharan commented on SPARK-3948: Damn, this sucks : the transferT

[jira] [Comment Edited] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173489#comment-14173489 ] Mridul Muralidharan edited comment on SPARK-3948 at 10/16/14 7:39 AM: --

[jira] [Commented] (SPARK-3814) Bitwise & does not work in Hive

2014-10-16 Thread Ravindra Pesala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173526#comment-14173526 ] Ravindra Pesala commented on SPARK-3814: Added support for Bitwise AND(&), OR(|) ,

[jira] [Updated] (SPARK-3814) Support for Bitwise AND(&), OR(|) ,XOR(^), NOT(~) in Spark HQL and SQL

2014-10-16 Thread Ravindra Pesala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala updated SPARK-3814: --- Summary: Support for Bitwise AND(&), OR(|) ,XOR(^), NOT(~) in Spark HQL and SQL (was: Bitwise

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173528#comment-14173528 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], Thanks a lot for your suggestio

[jira] [Commented] (SPARK-2706) Enable Spark to support Hive 0.13

2014-10-16 Thread qiaohaijun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173541#comment-14173541 ] qiaohaijun commented on SPARK-2706: --- sh make-distribution.sh --tgz -Phadoop-provided -P

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-16 Thread Praveen Seluka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173542#comment-14173542 ] Praveen Seluka commented on SPARK-3174: --- [~vanzin] Regarding your question related t

[jira] [Commented] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed

2014-10-16 Thread qiaohaijun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173543#comment-14173543 ] qiaohaijun commented on SPARK-1479: --- sh make-distribution.sh --tgz -Phadoop-provided -P

[jira] [Comment Edited] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-16 Thread Praveen Seluka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173542#comment-14173542 ] Praveen Seluka edited comment on SPARK-3174 at 10/16/14 8:55 AM: ---

[jira] [Comment Edited] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-16 Thread Praveen Seluka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173542#comment-14173542 ] Praveen Seluka edited comment on SPARK-3174 at 10/16/14 8:56 AM: ---

[jira] [Created] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-16 Thread JIRA
Christophe PRÉAUD created SPARK-3967: Summary: Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions Key: SPARK-3967

[jira] [Updated] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-16 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christophe PRÉAUD updated SPARK-3967: - Attachment: spark-1.1.0-yarn_cluster_tmpdir.patch Ensure that the temporary file which the

[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-16 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173566#comment-14173566 ] Christophe PRÉAUD commented on SPARK-3967: -- After investigating, it turns out tha

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173618#comment-14173618 ] Sean Owen commented on SPARK-3431: -- Yes that should be what scalatest does. It is a fork

[jira] [Commented] (SPARK-2321) Design a proper progress reporting & event listener API

2014-10-16 Thread Dev Lakhani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173619#comment-14173619 ] Dev Lakhani commented on SPARK-2321: There are some issues and bugs under the webui co

[jira] [Created] (SPARK-3968) Using parquet-mr filter2 api in spark sql, add a custom filter for InSet clause

2014-10-16 Thread Yash Datta (JIRA)
Yash Datta created SPARK-3968: - Summary: Using parquet-mr filter2 api in spark sql, add a custom filter for InSet clause Key: SPARK-3968 URL: https://issues.apache.org/jira/browse/SPARK-3968 Project: Spar

[jira] [Updated] (SPARK-3968) Using parquet-mr filter2 api in spark sql, add a custom filter for InSet clause

2014-10-16 Thread Yash Datta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Datta updated SPARK-3968: -- Shepherd: Yash Datta > Using parquet-mr filter2 api in spark sql, add a custom filter for InSet > claus

[jira] [Commented] (SPARK-3629) Improvements to YARN doc

2014-10-16 Thread ssj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173731#comment-14173731 ] ssj commented on SPARK-3629: need someone to verifity this patch > Improvements to YARN doc >

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173757#comment-14173757 ] Apache Spark commented on SPARK-3948: - User 'jerryshao' has created a pull request for

[jira] [Commented] (SPARK-3907) add "truncate table" support

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173806#comment-14173806 ] Apache Spark commented on SPARK-3907: - User 'wangxiaojing' has created a pull request

[jira] [Commented] (SPARK-3969) Optimizer should have a super class as an interface.

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173854#comment-14173854 ] Apache Spark commented on SPARK-3969: - User 'ueshin' has created a pull request for th

[jira] [Created] (SPARK-3970) Remove duplicate removal of local dirs

2014-10-16 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-3970: -- Summary: Remove duplicate removal of local dirs Key: SPARK-3970 URL: https://issues.apache.org/jira/browse/SPARK-3970 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-3969) Optimizer should have a super class as an interface.

2014-10-16 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3969: Summary: Optimizer should have a super class as an interface. Key: SPARK-3969 URL: https://issues.apache.org/jira/browse/SPARK-3969 Project: Spark Issue Type

[jira] [Commented] (SPARK-3970) Remove duplicate removal of local dirs

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173856#comment-14173856 ] Apache Spark commented on SPARK-3970: - User 'viirya' has created a pull request for th

[jira] [Commented] (SPARK-2365) Add IndexedRDD, an efficient updatable key-value store

2014-10-16 Thread Akshat Aranya (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173904#comment-14173904 ] Akshat Aranya commented on SPARK-2365: -- This looks great! I have been using IndexedR

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Dev Lakhani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173926#comment-14173926 ] Dev Lakhani commented on SPARK-3957: Here is my thoughts on a possible approach. Hi A

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173951#comment-14173951 ] Marcelo Vanzin commented on SPARK-3174: --- bq. Lets say we start the Spark applicatio

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173954#comment-14173954 ] Shivaram Venkataraman commented on SPARK-3957: -- I think it needs to be tracke

[jira] [Commented] (SPARK-3883) Provide SSL support for Akka and HttpServer based connections

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174005#comment-14174005 ] Marcelo Vanzin commented on SPARK-3883: --- FYI, any PR here should make sure the defau

[jira] [Commented] (SPARK-2750) Add Https support for Web UI

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174003#comment-14174003 ] Marcelo Vanzin commented on SPARK-2750: --- FYI, any PR here should make sure the defau

[jira] [Comment Edited] (SPARK-2750) Add Https support for Web UI

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174003#comment-14174003 ] Marcelo Vanzin edited comment on SPARK-2750 at 10/16/14 5:35 PM: ---

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174019#comment-14174019 ] Andrew Or commented on SPARK-3957: -- Yeah my understanding is that broadcast blocks aren't

[jira] [Closed] (SPARK-1761) Add broadcast information on SparkUI storage tab

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1761. Resolution: Duplicate > Add broadcast information on SparkUI storage tab > -

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174024#comment-14174024 ] Andrew Or commented on SPARK-3957: -- Hey [~devl.development] are you planning to work on t

[jira] [Commented] (SPARK-1761) Add broadcast information on SparkUI storage tab

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174026#comment-14174026 ] Andrew Or commented on SPARK-1761: -- Closing in favor of SPARK-3957, which is more descrip

[jira] [Created] (SPARK-3971) Failed to deserialize Vector in cluster mode

2014-10-16 Thread Davies Liu (JIRA)
Davies Liu created SPARK-3971: - Summary: Failed to deserialize Vector in cluster mode Key: SPARK-3971 URL: https://issues.apache.org/jira/browse/SPARK-3971 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2585. --- Resolution: Fixed Due to the CONFIGURATION_INSTANTIATION_LOCK thread-safety issue, I think that we'll

[jira] [Created] (SPARK-3972) PySpark Error on Windows with sc.wholeTextFiles

2014-10-16 Thread Michael Griffiths (JIRA)
Michael Griffiths created SPARK-3972: Summary: PySpark Error on Windows with sc.wholeTextFiles Key: SPARK-3972 URL: https://issues.apache.org/jira/browse/SPARK-3972 Project: Spark Issue T

[jira] [Commented] (SPARK-3736) Workers should reconnect to Master if disconnected

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174049#comment-14174049 ] Apache Spark commented on SPARK-3736: - User 'mccheah' has created a pull request for t

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174053#comment-14174053 ] Nan Zhu commented on SPARK-3957: I agree with [~andrewor14], I was also thinking about pig

[jira] [Created] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-3973: Summary: Print callSite information for broadcast variables Key: SPARK-3973 URL: https://issues.apache.org/jira/browse/SPARK-3973 Project: Spark

[jira] [Commented] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174066#comment-14174066 ] Apache Spark commented on SPARK-3973: - User 'shivaram' has created a pull request for

[jira] [Commented] (SPARK-2706) Enable Spark to support Hive 0.13

2014-10-16 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174073#comment-14174073 ] Zhan Zhang commented on SPARK-2706: --- The code does not go to upstream yet. To build 0.13

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Dev Lakhani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174107#comment-14174107 ] Dev Lakhani commented on SPARK-3957: Hi For now I am happy for [~CodingCat] to take

[jira] [Commented] (SPARK-3972) PySpark Error on Windows with sc.wholeTextFiles

2014-10-16 Thread Michael Griffiths (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174110#comment-14174110 ] Michael Griffiths commented on SPARK-3972: -- This issue does NOT occur if I build

[jira] [Updated] (SPARK-3882) JobProgressListener gets permanently out of sync with long running job

2014-10-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3882: --- Description: A long running spark context (non-streaming) will eventually start throwing the followin

[jira] [Created] (SPARK-3974) Block matrix abstracitons and partitioners

2014-10-16 Thread Reza Zadeh (JIRA)
Reza Zadeh created SPARK-3974: - Summary: Block matrix abstracitons and partitioners Key: SPARK-3974 URL: https://issues.apache.org/jira/browse/SPARK-3974 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-3975) Block Matrix addition and multiplication

2014-10-16 Thread Reza Zadeh (JIRA)
Reza Zadeh created SPARK-3975: - Summary: Block Matrix addition and multiplication Key: SPARK-3975 URL: https://issues.apache.org/jira/browse/SPARK-3975 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-3976) Detect block matrix partitioning schemes

2014-10-16 Thread Reza Zadeh (JIRA)
Reza Zadeh created SPARK-3976: - Summary: Detect block matrix partitioning schemes Key: SPARK-3976 URL: https://issues.apache.org/jira/browse/SPARK-3976 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-3977) Conversions between {Row, Coordinate}Matrix <-> BlockMatrix

2014-10-16 Thread Reza Zadeh (JIRA)
Reza Zadeh created SPARK-3977: - Summary: Conversions between {Row, Coordinate}Matrix <-> BlockMatrix Key: SPARK-3977 URL: https://issues.apache.org/jira/browse/SPARK-3977 Project: Spark Issue Ty

[jira] [Commented] (SPARK-3971) Failed to deserialize Vector in cluster mode

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174146#comment-14174146 ] Apache Spark commented on SPARK-3971: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action

2014-10-16 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174148#comment-14174148 ] Matt Cheah commented on SPARK-3466: --- I'll look into this. Someone please assign to me!

[jira] [Created] (SPARK-3978) Schema change on Spark-Hive (Parquet file format) table not working

2014-10-16 Thread Nilesh Barge (JIRA)
Nilesh Barge created SPARK-3978: --- Summary: Schema change on Spark-Hive (Parquet file format) table not working Key: SPARK-3978 URL: https://issues.apache.org/jira/browse/SPARK-3978 Project: Spark

[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action

2014-10-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3466: -- Assignee: Matthew Cheah > Limit size of results that a driver collects for each action > ---

[jira] [Updated] (SPARK-3971) Failed to deserialize Vector in cluster mode

2014-10-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3971: - Assignee: Davies Liu > Failed to deserialize Vector in cluster mode >

[jira] [Updated] (SPARK-3971) Failed to deserialize Vector in cluster mode

2014-10-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3971: - Target Version/s: 1.2.0 Affects Version/s: 1.2.0 > Failed to deserialize Vector in cluster mo

[jira] [Created] (SPARK-3979) Yarn backend's default file replication should match HDFS's default one

2014-10-16 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3979: - Summary: Yarn backend's default file replication should match HDFS's default one Key: SPARK-3979 URL: https://issues.apache.org/jira/browse/SPARK-3979 Project: Spar

[jira] [Commented] (SPARK-3979) Yarn backend's default file replication should match HDFS's default one

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174165#comment-14174165 ] Marcelo Vanzin commented on SPARK-3979: --- BTW, this would avoid issues like this: {n

[jira] [Updated] (SPARK-3979) Yarn backend's default file replication should match HDFS's default one

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-3979: -- Description: This code in ClientBase.scala sets the replication used for files uploaded to HDFS

[jira] [Commented] (SPARK-3979) Yarn backend's default file replication should match HDFS's default one

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174224#comment-14174224 ] Apache Spark commented on SPARK-3979: - User 'vanzin' has created a pull request for th

[jira] [Created] (SPARK-3980) GraphX Performance Issue

2014-10-16 Thread Jarred Li (JIRA)
Jarred Li created SPARK-3980: Summary: GraphX Performance Issue Key: SPARK-3980 URL: https://issues.apache.org/jira/browse/SPARK-3980 Project: Spark Issue Type: Bug Components: GraphX

[jira] [Created] (SPARK-3981) Consider a better approach to initialize SerDe on executors

2014-10-16 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-3981: Summary: Consider a better approach to initialize SerDe on executors Key: SPARK-3981 URL: https://issues.apache.org/jira/browse/SPARK-3981 Project: Spark Is

[jira] [Resolved] (SPARK-3971) Failed to deserialize Vector in cluster mode

2014-10-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-3971. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2830 [https://githu

[jira] [Updated] (SPARK-3980) GraphX Performance Issue

2014-10-16 Thread Jarred Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarred Li updated SPARK-3980: - Description: I run 4 workes in AWS (c3.xlarge), 4g memory for executor, 85,331,846 edges from(http://soci

[jira] [Created] (SPARK-3982) receiverStream in Python API

2014-10-16 Thread Davies Liu (JIRA)
Davies Liu created SPARK-3982: - Summary: receiverStream in Python API Key: SPARK-3982 URL: https://issues.apache.org/jira/browse/SPARK-3982 Project: Spark Issue Type: New Feature Compon

[jira] [Updated] (SPARK-3980) GraphX Performance Issue

2014-10-16 Thread Jarred Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarred Li updated SPARK-3980: - Description: I run 4 workes in AWS (c3.xlarge), 4g memory for executor, 85,331,846 edges from(http://soci

[jira] [Created] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-3983: - Summary: Scheduler delay (shown in the UI) is incorrect Key: SPARK-3983 URL: https://issues.apache.org/jira/browse/SPARK-3983 Project: Spark Issue Type: Bu

[jira] [Issue Comment Deleted] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-3983: -- Comment: was deleted (was: This is especially problematic when debugging performance of short ta

[jira] [Commented] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174452#comment-14174452 ] Kay Ousterhout commented on SPARK-3983: --- This is especially problematic when debuggi

[jira] [Updated] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-3983: -- Description: The reported scheduler delay includes time to get a new thread (from a threadpool)

[jira] [Comment Edited] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174507#comment-14174507 ] Marcelo Vanzin edited comment on SPARK-3877 at 10/17/14 12:08 AM: --

[jira] [Commented] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174507#comment-14174507 ] Marcelo Vanzin commented on SPARK-3877: --- [~tgraves] this can be seen as a subset of

[jira] [Created] (SPARK-3984) Display finer grained metrics about task launch overhead in the UI

2014-10-16 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-3984: - Summary: Display finer grained metrics about task launch overhead in the UI Key: SPARK-3984 URL: https://issues.apache.org/jira/browse/SPARK-3984 Project: Spark

[jira] [Commented] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174574#comment-14174574 ] Kay Ousterhout commented on SPARK-3983: --- https://github.com/apache/spark/pull/2832

[jira] [Issue Comment Deleted] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-3983: -- Comment: was deleted (was: https://github.com/apache/spark/pull/2832) > Scheduler delay (shown

[jira] [Updated] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-3983: -- Component/s: Web UI > Scheduler delay (shown in the UI) is incorrect > -

[jira] [Issue Comment Deleted] (SPARK-3984) Display finer grained metrics about task launch overhead in the UI

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-3984: -- Comment: was deleted (was: https://github.com/apache/spark/pull/2832) > Display finer grained m

[jira] [Commented] (SPARK-3984) Display finer grained metrics about task launch overhead in the UI

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174576#comment-14174576 ] Apache Spark commented on SPARK-3984: - User 'kayousterhout' has created a pull request

[jira] [Commented] (SPARK-3983) Scheduler delay (shown in the UI) is incorrect

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174577#comment-14174577 ] Apache Spark commented on SPARK-3983: - User 'kayousterhout' has created a pull request

[jira] [Commented] (SPARK-3984) Display finer grained metrics about task launch overhead in the UI

2014-10-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174575#comment-14174575 ] Kay Ousterhout commented on SPARK-3984: --- https://github.com/apache/spark/pull/2832

[jira] [Commented] (SPARK-3982) receiverStream in Python API

2014-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174580#comment-14174580 ] Apache Spark commented on SPARK-3982: - User 'davies' has created a pull request for th

[jira] [Resolved] (SPARK-3874) Provide stable TaskContext API

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3874. Resolution: Fixed Fix Version/s: 1.2.0 > Provide stable TaskContext API > ---

[jira] [Updated] (SPARK-3975) Block Matrix addition and multiplication

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3975: --- Component/s: MLlib > Block Matrix addition and multiplication > --

[jira] [Updated] (SPARK-3974) Block matrix abstracitons and partitioners

2014-10-16 Thread Reza Zadeh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Zadeh updated SPARK-3974: -- Component/s: MLlib > Block matrix abstracitons and partitioners > --

[jira] [Updated] (SPARK-3976) Detect block matrix partitioning schemes

2014-10-16 Thread Reza Zadeh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Zadeh updated SPARK-3976: -- Component/s: MLlib > Detect block matrix partitioning schemes >

[jira] [Updated] (SPARK-3977) Conversions between {Row, Coordinate}Matrix <-> BlockMatrix

2014-10-16 Thread Reza Zadeh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Zadeh updated SPARK-3977: -- Component/s: MLlib > Conversions between {Row, Coordinate}Matrix <-> BlockMatrix > -

[jira] [Commented] (SPARK-3882) JobProgressListener gets permanently out of sync with long running job

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174629#comment-14174629 ] Patrick Wendell commented on SPARK-3882: This is a known issue (SPARK-2316) that w

[jira] [Updated] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3973: --- Component/s: Spark Core > Print callSite information for broadcast variables > ---

[jira] [Closed] (SPARK-3923) All Standalone Mode services time out with each other

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3923. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Aaron Davidson Target Version/s

[jira] [Closed] (SPARK-3941) _remainingMem should not increase twice when updateBlockInfo

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3941. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Zhang, Liye Target Version/s: 1

[jira] [Updated] (SPARK-3890) remove redundant spark.executor.memory in doc

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3890: - Affects Version/s: (was: 1.2.0) 1.1.0 > remove redundant spark.executor.memory

[jira] [Closed] (SPARK-3890) remove redundant spark.executor.memory in doc

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3890. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee: WangTao

[jira] [Updated] (SPARK-3890) remove redundant spark.executor.memory in doc

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3890: - Affects Version/s: 1.2.0 > remove redundant spark.executor.memory in doc > ---

[jira] [Commented] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174660#comment-14174660 ] Patrick Wendell commented on SPARK-3963: In the initial version of this - I don't

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174662#comment-14174662 ] Josh Rosen commented on SPARK-3958: --- [~davies] ran across this exception while testing a

[jira] [Comment Edited] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174660#comment-14174660 ] Patrick Wendell edited comment on SPARK-3963 at 10/17/14 2:30 AM: --

[jira] [Updated] (SPARK-3067) JobProgressPage could not show Fair Scheduler Pools section sometimes

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3067: - Affects Version/s: 1.2.0 > JobProgressPage could not show Fair Scheduler Pools section sometimes > ---

[jira] [Closed] (SPARK-3067) JobProgressPage could not show Fair Scheduler Pools section sometimes

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3067. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: YanTang Zhai > JobProgressPage could not sh

[jira] [Updated] (SPARK-3067) JobProgressPage could not show Fair Scheduler Pools section sometimes

2014-10-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3067: - Affects Version/s: (was: 1.2.0) 1.1.0 > JobProgressPage could not show Fair Sch

  1   2   >