[jira] [Assigned] (SPARK-8426) Add blacklist mechanism for YARN container allocation

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-8426: --- Assignee: Apache Spark > Add blacklist mechanism for YARN container allocation >

[jira] [Assigned] (SPARK-8426) Add blacklist mechanism for YARN container allocation

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-8426: --- Assignee: (was: Apache Spark) > Add blacklist mechanism for YARN container allocation > -

[jira] [Commented] (SPARK-8426) Add blacklist mechanism for YARN container allocation

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744952#comment-14744952 ] Apache Spark commented on SPARK-8426: - User 'mwws' has created a pull request for this

[jira] [Resolved] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-10598. - Resolution: Fixed Fix Version/s: 1.6.0 > RoutingTablePartition toMessage method refers to

[jira] [Commented] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744931#comment-14744931 ] Reynold Xin commented on SPARK-10590: - Ok glad it worked! > Spark with YARN build i

[jira] [Commented] (SPARK-10466) UnsafeRow exception in Sort-Based Shuffle with data spill

2015-09-14 Thread Nalia Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744929#comment-14744929 ] Nalia Tang commented on SPARK-10466: 不知道这里适不适合提问。。也不知道中文是否恰当。。 在我使用decode 的时候 程序总是报:

[jira] [Commented] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Kevin Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744927#comment-14744927 ] Kevin Tsai commented on SPARK-10590: Hi Sean, Reynold, Thanks, It's my bad. This PR

[jira] [Created] (SPARK-10610) Using AppName instead AppId in the name of all metrics

2015-09-14 Thread Yi Tian (JIRA)
Yi Tian created SPARK-10610: --- Summary: Using AppName instead AppId in the name of all metrics Key: SPARK-10610 URL: https://issues.apache.org/jira/browse/SPARK-10610 Project: Spark Issue Type: New

[jira] [Created] (SPARK-10609) Improve task distribution strategy in taskSetManager

2015-09-14 Thread Zhang, Liye (JIRA)
Zhang, Liye created SPARK-10609: --- Summary: Improve task distribution strategy in taskSetManager Key: SPARK-10609 URL: https://issues.apache.org/jira/browse/SPARK-10609 Project: Spark Issue Type

[jira] [Created] (SPARK-10608) turn off reduce tasks locality as default to avoid bad cases

2015-09-14 Thread Zhang, Liye (JIRA)
Zhang, Liye created SPARK-10608: --- Summary: turn off reduce tasks locality as default to avoid bad cases Key: SPARK-10608 URL: https://issues.apache.org/jira/browse/SPARK-10608 Project: Spark I

[jira] [Commented] (SPARK-10486) Spark intermittently fails to recover from a worker failure (in standalone mode)

2015-09-14 Thread Madhusudanan Kandasamy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744858#comment-14744858 ] Madhusudanan Kandasamy commented on SPARK-10486: Can you share a simplifi

[jira] [Updated] (SPARK-10600) SparkSQL - Support for Not Exists in a Correlated Subquery

2015-09-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-10600: Component/s: SQL > SparkSQL - Support for Not Exists in a Correlated Subquery > ---

[jira] [Updated] (SPARK-10601) Spark SQL - Support for MINUS

2015-09-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-10601: Component/s: SQL > Spark SQL - Support for MINUS > - > >

[jira] [Resolved] (SPARK-10275) Add @since annotation to pyspark.mllib.random

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10275. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8666 [https://gi

[jira] [Resolved] (SPARK-10273) Add @since annotation to pyspark.mllib.feature

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10273. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8633 [https://gi

[jira] [Updated] (SPARK-9962) Decision Tree training: prevNodeIdsForInstances.unpersist() at end of training

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-9962: - Assignee: holdenk > Decision Tree training: prevNodeIdsForInstances.unpersist() at end of training

[jira] [Updated] (SPARK-9962) Decision Tree training: prevNodeIdsForInstances.unpersist() at end of training

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-9962: - Shepherd: Joseph K. Bradley Target Version/s: 1.6.0 > Decision Tree training: prevNode

[jira] [Resolved] (SPARK-9793) PySpark DenseVector, SparseVector should override __eq__ and __hash__

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-9793. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8166 [https://githu

[jira] [Commented] (SPARK-8939) YARN EC2 default setting fails with IllegalArgumentException

2015-09-14 Thread Sen Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744792#comment-14744792 ] Sen Fang commented on SPARK-8939: - The issue seems to lie in: https://github.com/amplab/s

[jira] [Commented] (SPARK-10194) SGD algorithms need convergenceTol parameter in Python

2015-09-14 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744782#comment-14744782 ] Yanbo Liang commented on SPARK-10194: - [~mengxr] OK. > SGD algorithms need convergen

[jira] [Comment Edited] (SPARK-8939) YARN EC2 default setting fails with IllegalArgumentException

2015-09-14 Thread Heji Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739578#comment-14739578 ] Heji Kim edited comment on SPARK-8939 at 9/15/15 3:14 AM: -- I was

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-09-14 Thread Sean McKibben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744746#comment-14744746 ] Sean McKibben commented on SPARK-6235: -- When reading from HBase into spark, the regio

[jira] [Resolved] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10542. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Target Version/s

[jira] [Commented] (SPARK-9313) Enable a "docker run" invocation in place of PYSPARK_PYTHON

2015-09-14 Thread Justin Uang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744731#comment-14744731 ] Justin Uang commented on SPARK-9313: This would be hugely helpful. I'm working on a pl

[jira] [Created] (SPARK-10607) Scheduler should include defensive measures against infinite loops due to task commit denial

2015-09-14 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-10607: -- Summary: Scheduler should include defensive measures against infinite loops due to task commit denial Key: SPARK-10607 URL: https://issues.apache.org/jira/browse/SPARK-10607

[jira] [Resolved] (SPARK-9851) Support submitting map stages individually in DAGScheduler

2015-09-14 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia resolved SPARK-9851. -- Resolution: Fixed Fix Version/s: 1.6.0 > Support submitting map stages individually in DA

[jira] [Commented] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Kevin Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744668#comment-14744668 ] Kevin Tsai commented on SPARK-10590: Hi Reynold, The issue still there to me. I've se

[jira] [Commented] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Kevin Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744660#comment-14744660 ] Kevin Tsai commented on SPARK-10590: Yes, I've ran ./dev/change-scala-version.sh 2.11

[jira] [Commented] (SPARK-10587) In pyspark, toDF() dosen't exsist in RDD object

2015-09-14 Thread SemiCoder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744643#comment-14744643 ] SemiCoder commented on SPARK-10587: --- It's not my code, it's code in latest released ver

[jira] [Comment Edited] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

2015-09-14 Thread Paul Wais (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744634#comment-14744634 ] Paul Wais edited comment on SPARK-10399 at 9/15/15 1:16 AM: A

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

2015-09-14 Thread Paul Wais (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744634#comment-14744634 ] Paul Wais commented on SPARK-10399: --- After investigating this issue a bit further, it m

[jira] [Created] (SPARK-10606) Cube/Rollup/GrpSet doesn't create the correct plan when group by is on something other than an AttributeReference

2015-09-14 Thread Harish Butani (JIRA)
Harish Butani created SPARK-10606: - Summary: Cube/Rollup/GrpSet doesn't create the correct plan when group by is on something other than an AttributeReference Key: SPARK-10606 URL: https://issues.apache.org/jira/b

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-09-14 Thread Ram Gande (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744605#comment-14744605 ] Ram Gande commented on SPARK-6235: -- Any progress on this. We are seeing this issue const

[jira] [Commented] (SPARK-9844) File appender race condition during SparkWorker shutdown

2015-09-14 Thread Gareth Lewin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744590#comment-14744590 ] Gareth Lewin commented on SPARK-9844: - I believe this still exists in 1.5.0, I am gett

[jira] [Created] (SPARK-10605) collect_list() and collect_set() should accept struct types as argument

2015-09-14 Thread Mike Fang (JIRA)
Mike Fang created SPARK-10605: - Summary: collect_list() and collect_set() should accept struct types as argument Key: SPARK-10605 URL: https://issues.apache.org/jira/browse/SPARK-10605 Project: Spark

[jira] [Created] (SPARK-10604) Univariate statistics as UDAFs: categorical stats

2015-09-14 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-10604: - Summary: Univariate statistics as UDAFs: categorical stats Key: SPARK-10604 URL: https://issues.apache.org/jira/browse/SPARK-10604 Project: Spark I

[jira] [Created] (SPARK-10603) Univariate statistics as UDAFs: multi-pass continuous stats

2015-09-14 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-10603: - Summary: Univariate statistics as UDAFs: multi-pass continuous stats Key: SPARK-10603 URL: https://issues.apache.org/jira/browse/SPARK-10603 Project: Spark

[jira] [Updated] (SPARK-10591) False negative in QueryTest.checkAnswer

2015-09-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10591: --- Description: # For double and float, {{NaN == NaN}} is always {{false}} # {{checkAnswer}} doesn't han

[jira] [Created] (SPARK-10602) Univariate statistics as UDAFs: single-pass continuous stats

2015-09-14 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-10602: - Summary: Univariate statistics as UDAFs: single-pass continuous stats Key: SPARK-10602 URL: https://issues.apache.org/jira/browse/SPARK-10602 Project: Spark

[jira] [Commented] (SPARK-8418) Add single- and multi-value support to ML Transformers

2015-09-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744508#comment-14744508 ] Joseph K. Bradley commented on SPARK-8418: -- Apologies for being AWOL! I'd defini

[jira] [Assigned] (SPARK-10317) start-history-server.sh CLI parsing incompatible with HistoryServer's arg parsing

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10317: Assignee: Apache Spark > start-history-server.sh CLI parsing incompatible with HistoryServ

[jira] [Assigned] (SPARK-10317) start-history-server.sh CLI parsing incompatible with HistoryServer's arg parsing

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10317: Assignee: (was: Apache Spark) > start-history-server.sh CLI parsing incompatible with

[jira] [Commented] (SPARK-10317) start-history-server.sh CLI parsing incompatible with HistoryServer's arg parsing

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744451#comment-14744451 ] Apache Spark commented on SPARK-10317: -- User 'rekhajoshm' has created a pull request

[jira] [Created] (SPARK-10601) Spark SQL - Support for MINUS

2015-09-14 Thread Richard Garris (JIRA)
Richard Garris created SPARK-10601: -- Summary: Spark SQL - Support for MINUS Key: SPARK-10601 URL: https://issues.apache.org/jira/browse/SPARK-10601 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-10600) SparkSQL - Support for Not Exists in a Correlated Subquery

2015-09-14 Thread Richard Garris (JIRA)
Richard Garris created SPARK-10600: -- Summary: SparkSQL - Support for Not Exists in a Correlated Subquery Key: SPARK-10600 URL: https://issues.apache.org/jira/browse/SPARK-10600 Project: Spark

[jira] [Resolved] (SPARK-10543) Peak Execution Memory Quantile should be Per-task Basis

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10543. --- Resolution: Fixed Assignee: Sen Fang Fix Version/s: 1.5.1

[jira] [Resolved] (SPARK-10549) scala 2.11 spark on yarn with security - Repl doesn't work

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10549. --- Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Target Version/s:

[jira] [Resolved] (SPARK-10576) Move .java files out of src/main/scala

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10576. --- Resolution: Fixed Assignee: Sean Owen Fix Version/s: 1.6.0 Target Version

[jira] [Resolved] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10594. --- Resolution: Fixed Fix Version/s: 1.6.0 Target Version/s: 1.6.0 > ApplicationMaster "

[jira] [Updated] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10594: -- Assignee: Erick Tryzelaar > ApplicationMaster "--help" references the removed "--num-executors" option

[jira] [Resolved] (SPARK-9996) Create local nested loop join operator

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-9996. -- Resolution: Fixed Fix Version/s: 1.6.0 > Create local nested loop join operator > ---

[jira] [Resolved] (SPARK-9997) Create local Expand operator

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-9997. -- Resolution: Fixed Fix Version/s: 1.6.0 Target Version/s: 1.6.0 > Create local Expand op

[jira] [Updated] (SPARK-6981) [SQL] SparkPlanner and QueryExecution should be factored out from SQLContext

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6981: - Assignee: Edoardo Vacchi > [SQL] SparkPlanner and QueryExecution should be factored out from SQLContext >

[jira] [Updated] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10522: -- Assignee: Davies Liu > Nanoseconds part of Timestamp should be positive in parquet > --

[jira] [Commented] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Robin East (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744359#comment-14744359 ] Robin East commented on SPARK-10598: Apologies - have checked it out. You're referrin

[jira] [Resolved] (SPARK-6981) [SQL] SparkPlanner and QueryExecution should be factored out from SQLContext

2015-09-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-6981. - Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 6356 [https:/

[jira] [Updated] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10598: -- Assignee: Robin East > RoutingTablePartition toMessage method refers to bytes instead of bits > ---

[jira] [Assigned] (SPARK-10599) Decrease communication in BlockMatrix multiply and increase performance

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10599: Assignee: (was: Apache Spark) > Decrease communication in BlockMatrix multiply and inc

[jira] [Commented] (SPARK-10599) Decrease communication in BlockMatrix multiply and increase performance

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744322#comment-14744322 ] Apache Spark commented on SPARK-10599: -- User 'brkyvz' has created a pull request for

[jira] [Assigned] (SPARK-10599) Decrease communication in BlockMatrix multiply and increase performance

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10599: Assignee: Apache Spark > Decrease communication in BlockMatrix multiply and increase perfo

[jira] [Updated] (SPARK-10575) Wrap RDD.takeSample with scope

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10575: -- Assignee: Vinod KC > Wrap RDD.takeSample with scope > -- > >

[jira] [Updated] (SPARK-10575) Wrap RDD.takeSample with scope

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10575: -- Affects Version/s: 1.4.0 Target Version/s: 1.6.0 > Wrap RDD.takeSample with scope > --

[jira] [Updated] (SPARK-10599) Decrease communication in BlockMatrix multiply and increase performance

2015-09-14 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-10599: Description: The BlockMatrix multiply sends each block to all the corresponding columns of the rig

[jira] [Updated] (SPARK-10563) SparkContext's local properties should be cloned when inherited

2015-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10563: -- Target Version/s: 1.6.0, 1.5.1 (was: 1.6.0) > SparkContext's local properties should be cloned when in

[jira] [Created] (SPARK-10599) Decrease communication in BlockMatrix multiply and increase performance

2015-09-14 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-10599: --- Summary: Decrease communication in BlockMatrix multiply and increase performance Key: SPARK-10599 URL: https://issues.apache.org/jira/browse/SPARK-10599 Project: Spark

[jira] [Commented] (SPARK-10563) SparkContext's local properties should be cloned when inherited

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744291#comment-14744291 ] Apache Spark commented on SPARK-10563: -- User 'andrewor14' has created a pull request

[jira] [Resolved] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10522. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Issue resolved by pull reque

[jira] [Resolved] (SPARK-10587) In pyspark, toDF() dosen't exsist in RDD object

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10587. --- Resolution: Not A Problem > In pyspark, toDF() dosen't exsist in RDD object > ---

[jira] [Updated] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10598: -- Affects Version/s: (was: 1.4.0) Target Version/s: (was: 1.5.0) Priority: Trivial

[jira] [Updated] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10598: -- Description: (was: (Have a look at https://cwiki.apache.org/confluence/display/SPARK/Contributing+t

[jira] [Assigned] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10598: Assignee: (was: Apache Spark) > RoutingTablePartition toMessage method refers to bytes

[jira] [Commented] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744275#comment-14744275 ] Apache Spark commented on SPARK-10598: -- User 'insidedctm' has created a pull request

[jira] [Assigned] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10598: Assignee: Apache Spark > RoutingTablePartition toMessage method refers to bytes instead of

[jira] [Created] (SPARK-10598) RoutingTablePartition toMessage method refers to bytes instead of bits

2015-09-14 Thread Robin East (JIRA)
Robin East created SPARK-10598: -- Summary: RoutingTablePartition toMessage method refers to bytes instead of bits Key: SPARK-10598 URL: https://issues.apache.org/jira/browse/SPARK-10598 Project: Spark

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744254#comment-14744254 ] Davies Liu commented on SPARK-9325: --- I would -1 on this. I'm worried that once we have

[jira] [Assigned] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10593: -- Assignee: Davies Liu > sql lateral view same name gives wrong value >

[jira] [Assigned] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10593: Assignee: Apache Spark > sql lateral view same name gives wrong value > --

[jira] [Assigned] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10593: Assignee: (was: Apache Spark) > sql lateral view same name gives wrong value > ---

[jira] [Commented] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744242#comment-14744242 ] Apache Spark commented on SPARK-10593: -- User 'davies' has created a pull request for

[jira] [Assigned] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10594: Assignee: (was: Apache Spark) > ApplicationMaster "--help" references the removed "--n

[jira] [Commented] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744209#comment-14744209 ] Apache Spark commented on SPARK-10594: -- User 'erickt' has created a pull request for

[jira] [Assigned] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10594: Assignee: Apache Spark > ApplicationMaster "--help" references the removed "--num-executor

[jira] [Updated] (SPARK-10573) IndexToString transformSchema adds output field as DoubleType

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10573: -- Fix Version/s: 1.5.1 > IndexToString transformSchema adds output field as DoubleType >

[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly

2015-09-14 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744200#comment-14744200 ] Ryan Williams commented on SPARK-7169: -- [~jlewandowski] I assume [~jerryshao] is refe

[jira] [Updated] (SPARK-10094) Mark ML PySpark feature transformers as Experimental to match Scala

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10094: -- Target Version/s: 1.6.0 (was: 1.5.0) > Mark ML PySpark feature transformers as Experimental to

[jira] [Resolved] (SPARK-10573) IndexToString transformSchema adds output field as DoubleType

2015-09-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10573. --- Resolution: Fixed Fix Version/s: 1.6.0 > IndexToString transformSchema adds output fie

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744175#comment-14744175 ] Shivaram Venkataraman commented on SPARK-9325: -- Hmm not necessarily. If `df$n

[jira] [Created] (SPARK-10597) MultivariateOnlineSummarizer for weighted instances

2015-09-14 Thread DB Tsai (JIRA)
DB Tsai created SPARK-10597: --- Summary: MultivariateOnlineSummarizer for weighted instances Key: SPARK-10597 URL: https://issues.apache.org/jira/browse/SPARK-10597 Project: Spark Issue Type: New Fea

[jira] [Updated] (SPARK-10597) MultivariateOnlineSummarizer for weighted instances

2015-09-14 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-10597: Description: MultivariateOnlineSummarizer for weighted instances is implemented as private API for SPARK-7

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744170#comment-14744170 ] Reynold Xin commented on SPARK-9325: Do you want to support collect(df$Age + 1) ? >

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744167#comment-14744167 ] Shivaram Venkataraman commented on SPARK-9325: -- Just `collect` and maybe `hea

[jira] [Commented] (SPARK-7040) Explore receiver-less DStream for Flume

2015-09-14 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744162#comment-14744162 ] Tathagata Das commented on SPARK-7040: -- I am not sure how Direct API can be built for

[jira] [Resolved] (SPARK-7040) Explore receiver-less DStream for Flume

2015-09-14 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-7040. -- Resolution: Invalid > Explore receiver-less DStream for Flume >

[jira] [Updated] (SPARK-10539) Intersection Optimization is Wrong

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-10539: Assignee: Yijie Shen > Intersection Optimization is Wrong > -- > >

[jira] [Commented] (SPARK-10588) Saving a DataFrame containing only nulls to JSON doesn't work

2015-09-14 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744157#comment-14744157 ] Yin Huai commented on SPARK-10588: -- Right, that is better. > Saving a DataFrame contain

[jira] [Commented] (SPARK-10588) Saving a DataFrame containing only nulls to JSON doesn't work

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744149#comment-14744149 ] Reynold Xin commented on SPARK-10588: - I think a more proper fix is to write the sche

[jira] [Closed] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-10590. --- Resolution: Cannot Reproduce Closing this one for now. [~kevintsai] please continue to comment if you

[jira] [Updated] (SPARK-10590) Spark with YARN build is broken

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-10590: Target Version/s: (was: 1.5.0) > Spark with YARN build is broken > --

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744136#comment-14744136 ] Reynold Xin commented on SPARK-9325: Are you only going to add collect, or are you goi

[jira] [Commented] (SPARK-10594) ApplicationMaster "--help" references the removed "--num-executors" option

2015-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744127#comment-14744127 ] Sean Owen commented on SPARK-10594: --- Can you make a PR instead? we use github rather th

  1   2   3   >