[jira] [Created] (SPARK-6333) saveAsObjectFile support for compression codec

2015-03-13 Thread Deenar Toraskar (JIRA)
Deenar Toraskar created SPARK-6333: -- Summary: saveAsObjectFile support for compression codec Key: SPARK-6333 URL: https://issues.apache.org/jira/browse/SPARK-6333 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4819) Remove Guava's "Optional" from public API

2015-03-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361558#comment-14361558 ] Marcelo Vanzin commented on SPARK-4819: --- Java 8 has {{java.lang.Optional}} which loo

[jira] [Commented] (SPARK-6332) compute calibration curve for binary classifiers

2015-03-13 Thread Robert Dodier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361525#comment-14361525 ] Robert Dodier commented on SPARK-6332: -- I've opened a PR for this: https://github.com

[jira] [Commented] (SPARK-6332) compute calibration curve for binary classifiers

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361518#comment-14361518 ] Apache Spark commented on SPARK-6332: - User 'robert-dodier' has created a pull request

[jira] [Updated] (SPARK-6332) compute calibration curve for binary classifiers

2015-03-13 Thread Robert Dodier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Dodier updated SPARK-6332: - Summary: compute calibration curve for binary classifiers (was: cmpute calibration curve for bina

[jira] [Updated] (SPARK-1363) Add streaming support for Spark SQL module

2015-03-13 Thread Jason Dai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dai updated SPARK-1363: - Assignee: Saisai Shao > Add streaming support for Spark SQL module > -

[jira] [Commented] (SPARK-6331) New Spark Master URL is not picked up when streaming context is started from checkpoint

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361501#comment-14361501 ] Apache Spark commented on SPARK-6331: - User 'tdas' has created a pull request for this

[jira] [Created] (SPARK-6332) cmpute calibration curve for binary classifiers

2015-03-13 Thread Robert Dodier (JIRA)
Robert Dodier created SPARK-6332: Summary: cmpute calibration curve for binary classifiers Key: SPARK-6332 URL: https://issues.apache.org/jira/browse/SPARK-6332 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-3643) Add cluster-specific config settings to configuration page

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3643. -- Resolution: Fixed I know this is a little aggressive, but if the suggestion is to duplicate all of the

[jira] [Updated] (SPARK-6317) Interactive HIVE scala console is not starting

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6317: - Assignee: Vinod KC > Interactive HIVE scala console is not starting >

[jira] [Commented] (SPARK-5790) VertexRDD's won't zip properly for `diff` capability

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361319#comment-14361319 ] Apache Spark commented on SPARK-5790: - User 'brennonyork' has created a pull request f

[jira] [Resolved] (SPARK-6317) Interactive HIVE scala console is not starting

2015-03-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-6317. --- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5011 [https://github.com/

[jira] [Updated] (SPARK-6317) Interactive HIVE scala console is not starting

2015-03-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6317: -- Description: {{build/sbt hive/console}} is failing {noformat{ [info] Starting scala interpreter... [

[jira] [Updated] (SPARK-6317) Interactive HIVE scala console is not starting

2015-03-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6317: -- Description: {{build/sbt hive/console}} is failing {noformat} [info] Starting scala interpreter... [

[jira] [Resolved] (SPARK-6285) Duplicated code leads to errors

2015-03-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-6285. --- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5010 [https://github.com/

[jira] [Commented] (SPARK-6329) Minor doc changes for Mesos and TOC

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361228#comment-14361228 ] Apache Spark commented on SPARK-6329: - User 'brennonyork' has created a pull request f

[jira] [Commented] (SPARK-1301) Add UI elements to collapse "Aggregated Metrics by Executor" pane on stage page

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361204#comment-14361204 ] Apache Spark commented on SPARK-1301: - User 'shankervalipireddy' has created a pull re

[jira] [Created] (SPARK-6331) New Spark Master URL is not picked up when streaming context is started from checkpoint

2015-03-13 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-6331: Summary: New Spark Master URL is not picked up when streaming context is started from checkpoint Key: SPARK-6331 URL: https://issues.apache.org/jira/browse/SPARK-6331

[jira] [Commented] (SPARK-6330) newParquetRelation gets incorrect FileSystem

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361168#comment-14361168 ] Apache Spark commented on SPARK-6330: - User 'vlyubin' has created a pull request for t

[jira] [Created] (SPARK-6330) newParquetRelation gets incorrect FileSystem

2015-03-13 Thread Volodymyr Lyubinets (JIRA)
Volodymyr Lyubinets created SPARK-6330: -- Summary: newParquetRelation gets incorrect FileSystem Key: SPARK-6330 URL: https://issues.apache.org/jira/browse/SPARK-6330 Project: Spark Issue

[jira] [Created] (SPARK-6329) Minor doc changes for Mesos and TOC

2015-03-13 Thread Brennon York (JIRA)
Brennon York created SPARK-6329: --- Summary: Minor doc changes for Mesos and TOC Key: SPARK-6329 URL: https://issues.apache.org/jira/browse/SPARK-6329 Project: Spark Issue Type: Bug Com

[jira] [Created] (SPARK-6328) Python API for StreamingListener

2015-03-13 Thread Yifan Wang (JIRA)
Yifan Wang created SPARK-6328: - Summary: Python API for StreamingListener Key: SPARK-6328 URL: https://issues.apache.org/jira/browse/SPARK-6328 Project: Spark Issue Type: Improvement Co

[jira] [Commented] (SPARK-6327) Run PySpark with python directly is broken

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361104#comment-14361104 ] Apache Spark commented on SPARK-6327: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-6288) Pyrolite calls hashCode to cache previously serialized objects

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361037#comment-14361037 ] Xiangrui Meng commented on SPARK-6288: -- Test the following code: {code} from pyspark

[jira] [Created] (SPARK-6327) Run PySpark with python directly is broken

2015-03-13 Thread Davies Liu (JIRA)
Davies Liu created SPARK-6327: - Summary: Run PySpark with python directly is broken Key: SPARK-6327 URL: https://issues.apache.org/jira/browse/SPARK-6327 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-6282) Strange Python import error when using random() in a lambda function

2015-03-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361030#comment-14361030 ] Davies Liu commented on SPARK-6282: --- [~laskov] The following code runs fine here (master

[jira] [Comment Edited] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361005#comment-14361005 ] Debasish Das edited comment on SPARK-6323 at 3/13/15 7:48 PM: --

[jira] [Commented] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361005#comment-14361005 ] Debasish Das commented on SPARK-6323: - There are some other interesting cases for larg

[jira] [Updated] (SPARK-6315) SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1

2015-03-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-6315: Target Version/s: 1.3.1 (was: 1.3.0) > SparkSQL 1.3.0 (RC3) fails to read parquet file gene

[jira] [Commented] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360956#comment-14360956 ] Debasish Das commented on SPARK-6323: - g(z) is not regularization...we support constra

[jira] [Commented] (SPARK-6285) Duplicated code leads to errors

2015-03-13 Thread Iulian Dragos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360937#comment-14360937 ] Iulian Dragos commented on SPARK-6285: -- Thanks, [~lian cheng] > Duplicated code lead

[jira] [Updated] (SPARK-4600) org.apache.spark.graphx.VertexRDD.diff does not work

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4600: - Component/s: Documentation Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) >

[jira] [Resolved] (SPARK-4600) org.apache.spark.graphx.VertexRDD.diff does not work

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4600. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5015 [https://github.com/ap

[jira] [Updated] (SPARK-6275) Miss toDF() function in docs/sql-programming-guide.md

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6275: - Fix Version/s: 1.3.1 > Miss toDF() function in docs/sql-programming-guide.md > --

[jira] [Commented] (SPARK-6325) YarnAllocator crash with dynamic allocation on

2015-03-13 Thread Wing Yew Poon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360894#comment-14360894 ] Wing Yew Poon commented on SPARK-6325: -- The testcase actually came from a similar exa

[jira] [Commented] (SPARK-6325) YarnAllocator crash with dynamic allocation on

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360898#comment-14360898 ] Apache Spark commented on SPARK-6325: - User 'vanzin' has created a pull request for th

[jira] [Commented] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

2015-03-13 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360896#comment-14360896 ] Josh Rosen commented on SPARK-6313: --- Thanks for the pointer to the Lucene lock factory c

[jira] [Commented] (SPARK-6326) Improve castStruct to be faster

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360888#comment-14360888 ] Apache Spark commented on SPARK-6326: - User 'viirya' has created a pull request for th

[jira] [Updated] (SPARK-6133) SparkContext#stop is not idempotent

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6133: - Labels: (was: backport-needed) > SparkContext#stop is not idempotent > -

[jira] [Commented] (SPARK-6288) Pyrolite calls hashCode to cache previously serialized objects

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360883#comment-14360883 ] Xiangrui Meng commented on SPARK-6288: -- Yes, `memo` is a private variable. I sent a p

[jira] [Updated] (SPARK-6326) Improve castStruct to be faster

2015-03-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-6326: --- Summary: Improve castStruct to be faster (was: Make castStruct faster) > Improve castStruct t

[jira] [Resolved] (SPARK-6133) SparkContext#stop is not idempotent

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6133. -- Resolution: Fixed Fix Version/s: 1.3.1 > SparkContext#stop is not idempotent > --

[jira] [Created] (SPARK-6326) Make castStruct faster

2015-03-13 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-6326: -- Summary: Make castStruct faster Key: SPARK-6326 URL: https://issues.apache.org/jira/browse/SPARK-6326 Project: Spark Issue Type: Improvement Co

[jira] [Updated] (SPARK-6132) Context cleaner race condition across SparkContexts

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6132: - Fix Version/s: 1.3.1 Also in 1.3.1 now; Andrew mentioned he'd like to back-port after seeing how this goe

[jira] [Resolved] (SPARK-6087) Provide actionable exception if Kryo buffer is not large enough

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6087. -- Resolution: Fixed Fix Version/s: 1.3.1 Target Version/s: (was: 1.4.0, 1.3.1) > Provid

[jira] [Updated] (SPARK-6087) Provide actionable exception if Kryo buffer is not large enough

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6087: - Labels: starter (was: backport-needed starter) > Provide actionable exception if Kryo buffer is not large

[jira] [Commented] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

2015-03-13 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360842#comment-14360842 ] Josh Rosen commented on SPARK-6313: --- Could you update this ticket with more details on t

[jira] [Updated] (SPARK-6036) EventLog process logic has race condition with Akka actor system

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6036: - Labels: (was: backport-needed) > EventLog process logic has race condition with Akka actor system >

[jira] [Resolved] (SPARK-6036) EventLog process logic has race condition with Akka actor system

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6036. -- Resolution: Fixed Fix Version/s: 1.3.1 Target Version/s: (was: 1.4.0, 1.3.1) > EventL

[jira] [Created] (SPARK-6325) YarnAllocator crash with dynamic allocation on

2015-03-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-6325: - Summary: YarnAllocator crash with dynamic allocation on Key: SPARK-6325 URL: https://issues.apache.org/jira/browse/SPARK-6325 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-4044) Thriftserver fails to start when JAVA_HOME points to JRE instead of JDK

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360816#comment-14360816 ] Sean Owen commented on SPARK-4044: -- Note that this is also fixed, by being obsoleted, in

[jira] [Resolved] (SPARK-4044) Thriftserver fails to start when JAVA_HOME points to JRE instead of JDK

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4044. -- Resolution: Fixed Fix Version/s: 1.3.1 Issue resolved by pull request 4981 [https://github.com/ap

[jira] [Commented] (SPARK-6229) Support encryption in network/common module

2015-03-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360808#comment-14360808 ] Aaron Davidson commented on SPARK-6229: --- The reason we did not originally put the SA

[jira] [Updated] (SPARK-4300) Race condition during SparkWorker shutdown

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4300: - Labels: (was: backport-needed) > Race condition during SparkWorker shutdown > --

[jira] [Resolved] (SPARK-4300) Race condition during SparkWorker shutdown

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4300. -- Resolution: Fixed Fix Version/s: 1.3.1 Target Version/s: (was: 1.3.0, 1.2.2, 1.4.0) >

[jira] [Commented] (SPARK-6288) Pyrolite calls hashCode to cache previously serialized objects

2015-03-13 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360799#comment-14360799 ] Josh Rosen commented on SPARK-6288: --- Do we have to modify Pyrolite in order to disable t

[jira] [Commented] (SPARK-6288) Pyrolite calls hashCode to cache previously serialized objects

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360784#comment-14360784 ] Xiangrui Meng commented on SPARK-6288: -- [~joshrosen] The memoLookup cost is actually

[jira] [Updated] (SPARK-6288) Pyrolite calls hashCode to cache previously serialized objects

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-6288: - Attachment: Screen Shot 2015-03-13 at 10.45.35 AM.png Attached a screenshot of YourKit profiling r

[jira] [Updated] (SPARK-6194) collect() in PySpark will cause memory leak in JVM

2015-03-13 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-6194: -- Fix Version/s: 1.3.1 1.4.0 1.2.2 I've merged this into `master` (1

[jira] [Updated] (SPARK-4704) SparkSubmitDriverBootstrap doesn't flush output

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4704: - Labels: (was: backport-needed) > SparkSubmitDriverBootstrap doesn't flush output > -

[jira] [Resolved] (SPARK-4704) SparkSubmitDriverBootstrap doesn't flush output

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4704. -- Resolution: Fixed Fix Version/s: 1.3.1 Target Version/s: (was: 1.2.2, 1.4.0, 1.3.1) >

[jira] [Resolved] (SPARK-6252) Scala NaiveBayes should expose getLambda

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-6252. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 4969 [https://githu

[jira] [Resolved] (SPARK-6278) Mention the change of step size in the migration guide

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-6278. -- Resolution: Fixed Fix Version/s: 1.3.1 Issue resolved by pull request 4978 [https://githu

[jira] [Updated] (SPARK-6278) Mention the change of step size in the migration guide

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-6278: - Fix Version/s: 1.4.0 > Mention the change of step size in the migration guide > --

[jira] [Updated] (SPARK-4964) Exactly-once + WAL-free Kafka Support in Spark Streaming

2015-03-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4964: --- Assignee: Cody Koeninger > Exactly-once + WAL-free Kafka Support in Spark Streaming >

[jira] [Updated] (SPARK-6324) Clean up usage code in command-line scripts

2015-03-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-6324: -- Description: With SPARK-4924, most of the logic to launch Spark classes is in a new Java librar

[jira] [Created] (SPARK-6324) Clean up usage code in command-line scripts

2015-03-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-6324: - Summary: Clean up usage code in command-line scripts Key: SPARK-6324 URL: https://issues.apache.org/jira/browse/SPARK-6324 Project: Spark Issue Type: Impro

[jira] [Comment Edited] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360679#comment-14360679 ] Xiangrui Meng edited comment on SPARK-6323 at 3/13/15 5:13 PM: -

[jira] [Commented] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360679#comment-14360679 ] Xiangrui Meng commented on SPARK-6323: -- [~debasish83] Please help me understand some

[jira] [Assigned] (SPARK-3735) Sending the factor directly or AtA based on the cost in ALS

2015-03-13 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-3735: Assignee: Xiangrui Meng > Sending the factor directly or AtA based on the cost in ALS > ---

[jira] [Commented] (SPARK-5541) Allow running Maven or SBT in run-tests

2015-03-13 Thread Brennon York (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360628#comment-14360628 ] Brennon York commented on SPARK-5541: - [~nchammas], [~pwendell] was there anything spe

[jira] [Commented] (SPARK-5790) VertexRDD's won't zip properly for `diff` capability

2015-03-13 Thread Brennon York (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360613#comment-14360613 ] Brennon York commented on SPARK-5790: - [~maropu] did you get those tests in a PR or in

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360616#comment-14360616 ] Sean Owen commented on SPARK-4820: -- Most certainly worth a note at least, yes. Feel free

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread Theodore Vasiloudis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360610#comment-14360610 ] Theodore Vasiloudis commented on SPARK-4820: [~srowen] Including a warning in

[jira] [Commented] (SPARK-6095) Support model save/load in Python's linear models

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360599#comment-14360599 ] Apache Spark commented on SPARK-6095: - User 'yanboliang' has created a pull request fo

[jira] [Commented] (SPARK-4600) org.apache.spark.graphx.VertexRDD.diff does not work

2015-03-13 Thread Brennon York (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360570#comment-14360570 ] Brennon York commented on SPARK-4600: - Thanks for the clarification [~ankurd]! I just

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which scales t

[jira] [Commented] (SPARK-6095) Support model save/load in Python's linear models

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360569#comment-14360569 ] Apache Spark commented on SPARK-6095: - User 'yanboliang' has created a pull request fo

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which scales t

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Commented] (SPARK-4600) org.apache.spark.graphx.VertexRDD.diff does not work

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360562#comment-14360562 ] Apache Spark commented on SPARK-4600: - User 'brennonyork' has created a pull request f

[jira] [Commented] (SPARK-2344) Add Fuzzy C-Means algorithm to MLlib

2015-03-13 Thread Alex (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360560#comment-14360560 ] Alex commented on SPARK-2344: - I see, actually I was counting to make the implementation of th

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Resolved] (SPARK-4231) Add RankingMetrics to examples.MovieLensALS

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das resolved SPARK-4231. - Resolution: Duplicate > Add RankingMetrics to examples.MovieLensALS >

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Created] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
Debasish Das created SPARK-6323: --- Summary: Large rank matrix factorization with Nonlinear loss and constraints Key: SPARK-6323 URL: https://issues.apache.org/jira/browse/SPARK-6323 Project: Spark

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Updated] (SPARK-6323) Large rank matrix factorization with Nonlinear loss and constraints

2015-03-13 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debasish Das updated SPARK-6323: Description: Currently ml.recommendation.ALS is optimized for gram matrix generation which only sca

[jira] [Closed] (SPARK-6022) GraphX `diff` test incorrectly operating on values (not VertexId's)

2015-03-13 Thread Brennon York (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brennon York closed SPARK-6022. --- Resolution: Fixed Awesome, thanks for the clarity guys! I'll close this JIRA and introduce a new one

[jira] [Commented] (SPARK-6282) Strange Python import error when using random() in a lambda function

2015-03-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360534#comment-14360534 ] Nicholas Chammas commented on SPARK-6282: - [~joshrosen], [~davies]: Does this erro

[jira] [Commented] (SPARK-6322) CTAS should consider the case where no file format or storage handler is given

2015-03-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360485#comment-14360485 ] Apache Spark commented on SPARK-6322: - User 'viirya' has created a pull request for th

[jira] [Created] (SPARK-6322) CTAS should consider the case where no file format or storage handler is given

2015-03-13 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-6322: -- Summary: CTAS should consider the case where no file format or storage handler is given Key: SPARK-6322 URL: https://issues.apache.org/jira/browse/SPARK-6322 Proj

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360452#comment-14360452 ] Óscar Puertas commented on SPARK-4820: -- I have Ubuntu 14.04 but, you are right, the f

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360450#comment-14360450 ] Sean Owen commented on SPARK-4820: -- What's your environment / file system? > Spark build

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360443#comment-14360443 ] Óscar Puertas commented on SPARK-4820: -- I see, but I have no an encrypted file system

[jira] [Updated] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4820: - Priority: Minor (was: Major) Yes, it is not resolved. I am not sure it is worth modifying the build for

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2015-03-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360407#comment-14360407 ] Óscar Puertas commented on SPARK-4820: -- Still failing for the 1.3 branch. > Spark bu

[jira] [Updated] (SPARK-6299) ClassNotFoundException in standalone mode when running groupByKey with class defined in REPL.

2015-03-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6299: - Priority: Major (was: Critical) Summary: ClassNotFoundException in standalone mode when running group

[jira] [Commented] (SPARK-6315) SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1

2015-03-13 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360399#comment-14360399 ] Yin Huai commented on SPARK-6315: - Should we change the target version to 1.3.1? > SparkS

  1   2   >