[jira] [Resolved] (SPARK-2656) Python version without support for exact sample size

2014-07-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2656. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 1554 [https://gith

[jira] [Updated] (SPARK-2656) Python version without support for exact sample size

2014-07-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2656: - Fix Version/s: (was: 1.2.0) 1.1.0 > Python version without support for exa

[jira] [Commented] (SPARK-2686) Add Length support to Spark SQL and HQL and Strlen support to SQL

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074135#comment-14074135 ] Apache Spark commented on SPARK-2686: - User 'javadba' has created a pull request for t

[jira] [Commented] (SPARK-2683) unidoc failed because org.apache.spark.util.CallSite uses Java keywords as value names

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074129#comment-14074129 ] Apache Spark commented on SPARK-2683: - User 'yhuai' has created a pull request for thi

[jira] [Created] (SPARK-2686) Add Length support to Spark SQL and HQL and Strlen support to SQL

2014-07-24 Thread Stephen Boesch (JIRA)
Stephen Boesch created SPARK-2686: - Summary: Add Length support to Spark SQL and HQL and Strlen support to SQL Key: SPARK-2686 URL: https://issues.apache.org/jira/browse/SPARK-2686 Project: Spark

[jira] [Created] (SPARK-2685) Update ExternalAppendOnlyMap to avoid buffer.remove()

2014-07-24 Thread Matei Zaharia (JIRA)
Matei Zaharia created SPARK-2685: Summary: Update ExternalAppendOnlyMap to avoid buffer.remove() Key: SPARK-2685 URL: https://issues.apache.org/jira/browse/SPARK-2685 Project: Spark Issue Typ

[jira] [Created] (SPARK-2684) Update ExternalAppendOnlyMap to take an iterator as input

2014-07-24 Thread Matei Zaharia (JIRA)
Matei Zaharia created SPARK-2684: Summary: Update ExternalAppendOnlyMap to take an iterator as input Key: SPARK-2684 URL: https://issues.apache.org/jira/browse/SPARK-2684 Project: Spark Issue

[jira] [Created] (SPARK-2683) unidoc failed because org.apache.spark.util.CallSite uses Java keywords as value names

2014-07-24 Thread Yin Huai (JIRA)
Yin Huai created SPARK-2683: --- Summary: unidoc failed because org.apache.spark.util.CallSite uses Java keywords as value names Key: SPARK-2683 URL: https://issues.apache.org/jira/browse/SPARK-2683 Project: S

[jira] [Commented] (SPARK-2575) SVMWithSGD throwing Input Validation failed

2014-07-24 Thread navanee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074118#comment-14074118 ] navanee commented on SPARK-2575: yeah Thanks for Clarifying my doubt > SVMWithSGD throwin

[jira] [Resolved] (SPARK-2538) External aggregation in Python

2014-07-24 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia resolved SPARK-2538. -- Resolution: Fixed Fix Version/s: (was: 1.0.1) (was: 1.0.0)

[jira] [Commented] (SPARK-2664) Deal with `--conf` options in spark-submit that relate to flags

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074108#comment-14074108 ] Patrick Wendell commented on SPARK-2664: Hey Sandy, The reason why we originally

[jira] [Commented] (SPARK-2682) Javadoc generated from Scala source code is not in javadoc's index

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074109#comment-14074109 ] Apache Spark commented on SPARK-2682: - User 'yhuai' has created a pull request for thi

[jira] [Updated] (SPARK-2682) Javadoc generated from Scala source code is not in javadoc's index

2014-07-24 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-2682: Component/s: Documentation > Javadoc generated from Scala source code is not in javadoc's index > -

[jira] [Created] (SPARK-2682) Javadoc generated from Scala source code is not in javadoc's index

2014-07-24 Thread Yin Huai (JIRA)
Yin Huai created SPARK-2682: --- Summary: Javadoc generated from Scala source code is not in javadoc's index Key: SPARK-2682 URL: https://issues.apache.org/jira/browse/SPARK-2682 Project: Spark Issue

[jira] [Updated] (SPARK-2670) FetchFailedException should be thrown when local fetch has failed

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2670: --- Priority: Critical (was: Major) > FetchFailedException should be thrown when local fetch has

[jira] [Updated] (SPARK-2670) FetchFailedException should be thrown when local fetch has failed

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2670: --- Component/s: Spark Core > FetchFailedException should be thrown when local fetch has failed >

[jira] [Updated] (SPARK-2670) FetchFailedException should be thrown when local fetch has failed

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2670: --- Target Version/s: 1.1.0 > FetchFailedException should be thrown when local fetch has failed >

[jira] [Comment Edited] (SPARK-2681) Spark can hang when fetching shuffle blocks

2014-07-24 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074044#comment-14074044 ] Guoqiang Li edited comment on SPARK-2681 at 7/25/14 4:39 AM: -

[jira] [Commented] (SPARK-2618) use config spark.scheduler.priority for specifying TaskSet's priority on DAGScheduler

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074045#comment-14074045 ] Patrick Wendell commented on SPARK-2618: We shouldn't should expose these types of

[jira] [Commented] (SPARK-2681) Spark can hang when fetching shuffle blocks

2014-07-24 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074044#comment-14074044 ] Guoqiang Li commented on SPARK-2681: OK, but have some time. > Spark can hang when fe

[jira] [Commented] (SPARK-2681) With low probability, the Spark inexplicable hang

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074031#comment-14074031 ] Patrick Wendell commented on SPARK-2681: Can you do a jstack of the executor when

[jira] [Updated] (SPARK-2681) Spark can hang when fetching shuffle blocks

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2681: --- Description: executor log : {noformat} 14/07/24 22:56:52 INFO executor.CoarseGrainedExec

[jira] [Updated] (SPARK-2681) Spark can hang when fetching shuffle blocks

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2681: --- Component/s: Spark Core > Spark can hang when fetching shuffle blocks > -

[jira] [Updated] (SPARK-2681) Spark can hang when fetching shuffle blocks

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2681: --- Summary: Spark can hang when fetching shuffle blocks (was: With low probability, the Spark i

[jira] [Commented] (SPARK-1719) spark.executor.extraLibraryPath isn't applied on yarn

2014-07-24 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074026#comment-14074026 ] Guoqiang Li commented on SPARK-1719: Seems to be related. > spark.executor.extraLibr

[jira] [Commented] (SPARK-1719) spark.executor.extraLibraryPath isn't applied on yarn

2014-07-24 Thread Xuri Nagarin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074021#comment-14074021 ] Xuri Nagarin commented on SPARK-1719: - Is this related? When I run spark-shell with "-

[jira] [Created] (SPARK-2681) With low probability, the Spark inexplicable hang

2014-07-24 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-2681: -- Summary: With low probability, the Spark inexplicable hang Key: SPARK-2681 URL: https://issues.apache.org/jira/browse/SPARK-2681 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2515) Hypothesis testing

2014-07-24 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074003#comment-14074003 ] Hossein Falaki commented on SPARK-2515: --- If we really have to implement another chi-

[jira] [Commented] (SPARK-2529) Clean the closure in foreach and foreachPartition

2014-07-24 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074002#comment-14074002 ] Mark Hamstra commented on SPARK-2529: - Actually, we were cleaning those closures, but

[jira] [Commented] (SPARK-2387) Remove the stage barrier for better resource utilization

2014-07-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073989#comment-14073989 ] Rui Li commented on SPARK-2387: --- [~kayousterhout] Thanks for the review. I tested this PoC w

[jira] [Updated] (SPARK-2668) Add variable of yarn log directory for reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Description: Assign value of yarn container log directory to java opts "spark.yarn.log.dir", So user d

[jira] [Updated] (SPARK-2668) Add variable of yarn log directory for reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Summary: Add variable of yarn log directory for reference from the log4j configuration (was: Add varia

[jira] [Updated] (SPARK-2668) Add variable of yarn log directory to reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Summary: Add variable of yarn log directory to reference from the log4j configuration (was: Add variab

[jira] [Updated] (SPARK-2668) Add variable of yarn log diectory to reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Description: Assign value of yarn container log directory to java opts "spark.yarn.log.dir", So user d

[jira] [Updated] (SPARK-2668) Add variable of yarn log diectory to reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Description: Assign value of yarn container log directory to java opts "spark.yarn.log.dir", So user d

[jira] [Commented] (SPARK-2668) Add variable of yarn log diectory to reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073982#comment-14073982 ] Peng Zhang commented on SPARK-2668: --- I changed the title to make it more clear. And I f

[jira] [Updated] (SPARK-2668) Add variable of yarn log diectory to reference from the log4j configuration

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Summary: Add variable of yarn log diectory to reference from the log4j configuration (was: Support log

[jira] [Updated] (SPARK-2668) Support log4j log to yarn container log directory

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Affects Version/s: 1.0.0 > Support log4j log to yarn container log directory >

[jira] [Updated] (SPARK-2668) Support log4j log to yarn container log directory

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Zhang updated SPARK-2668: -- Fix Version/s: (was: 1.0.0) > Support log4j log to yarn container log directory > -

[jira] [Commented] (SPARK-2529) Clean the closure in foreach and foreachPartition

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073964#comment-14073964 ] Apache Spark commented on SPARK-2529: - User 'rxin' has created a pull request for this

[jira] [Updated] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2425: - Target Version/s: 1.0.3 (was: 1.0.2) > Standalone Master is too aggressive in removing Applicati

[jira] [Updated] (SPARK-2541) Standalone mode can't access secure HDFS anymore

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2541: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > Standalone mode can't access secure HDFS an

[jira] [Updated] (SPARK-1667) Jobs never finish successfully once bucket file missing occurred

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-1667: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > Jobs never finish successfully once bucket

[jira] [Updated] (SPARK-2558) Mention --queue argument in YARN documentation

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2558: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > Mention --queue argument in YARN documentat

[jira] [Updated] (SPARK-2576) slave node throws NoClassDefFoundError $line11.$read$ when executing a Spark QL query on HDFS CSV file

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2576: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > slave node throws NoClassDefFoundError $lin

[jira] [Updated] (SPARK-2506) In yarn-cluster mode, ApplicationMaster does not clean up correctly at the end of the job if users call sc.stop manually

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2506: - Target Version/s: 1.0.3 (was: 1.0.2) > In yarn-cluster mode, ApplicationMaster does not clean up

[jira] [Updated] (SPARK-2548) JavaRecoverableWordCount is missing

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2548: - Target Version/s: 1.1.0, 0.9.3, 1.0.3 (was: 1.1.0, 1.0.2, 0.9.3) > JavaRecoverableWordCount is m

[jira] [Updated] (SPARK-2529) Clean the closure in foreach and foreachPartition

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2529: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > Clean the closure in foreach and foreachPar

[jira] [Updated] (SPARK-2531) Make BroadcastNestedLoopJoin take into account a BuildSide

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2531: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0, 1.0.2) > Make BroadcastNestedLoopJoin take into acco

[jira] [Created] (SPARK-2680) Lower spark.shuffle.memoryFraction to 0.2 by default

2014-07-24 Thread Matei Zaharia (JIRA)
Matei Zaharia created SPARK-2680: Summary: Lower spark.shuffle.memoryFraction to 0.2 by default Key: SPARK-2680 URL: https://issues.apache.org/jira/browse/SPARK-2680 Project: Spark Issue Type

[jira] [Resolved] (SPARK-1030) unneeded file required when running pyspark program using yarn-client

2014-07-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-1030. --- Resolution: Fixed Fix Version/s: 1.0.0 Closing this now, since it was addressed as part of Spa

[jira] [Commented] (SPARK-1044) Default spark logs location in EC2 AMI leads to out-of-disk space pretty soon

2014-07-24 Thread Allan Douglas R. de Oliveira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073938#comment-14073938 ] Allan Douglas R. de Oliveira commented on SPARK-1044: - I think it is s

[jira] [Commented] (SPARK-786) Clean up old work directories in standalone worker

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073932#comment-14073932 ] Andrew Ash commented on SPARK-786: -- Agreed. With SPARK-1860 we could re-enable that the f

[jira] [Commented] (SPARK-1044) Default spark logs location in EC2 AMI leads to out-of-disk space pretty soon

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073930#comment-14073930 ] Andrew Ash commented on SPARK-1044: --- Filling up the work dir could be alleviated by fixi

[jira] [Resolved] (SPARK-2014) Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default

2014-07-24 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia resolved SPARK-2014. -- Resolution: Fixed Fix Version/s: 1.1.0 > Make PySpark store RDDs in MEMORY_ONLY_SER with

[jira] [Resolved] (SPARK-2464) Twitter Receiver does not stop correctly when streamingContext.stop is called

2014-07-24 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-2464. -- Resolution: Fixed > Twitter Receiver does not stop correctly when streamingContext.stop is call

[jira] [Commented] (SPARK-2515) Hypothesis testing

2014-07-24 Thread Doris Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073879#comment-14073879 ] Doris Xin commented on SPARK-2515: -- Here's the proposed API for chi-squared tests (lives

[jira] [Updated] (SPARK-2298) Show stage attempt in UI

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2298: --- Priority: Critical (was: Major) > Show stage attempt in UI > > >

[jira] [Commented] (SPARK-2679) Ser/De for Double to enable calling Java API from python in MLlib

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073833#comment-14073833 ] Apache Spark commented on SPARK-2679: - User 'dorx' has created a pull request for this

[jira] [Created] (SPARK-2679) Ser/De for Double to enable calling Java API from python in MLlib

2014-07-24 Thread Doris Xin (JIRA)
Doris Xin created SPARK-2679: Summary: Ser/De for Double to enable calling Java API from python in MLlib Key: SPARK-2679 URL: https://issues.apache.org/jira/browse/SPARK-2679 Project: Spark Issu

[jira] [Updated] (SPARK-2678) `Spark-submit` overrides user application options

2014-07-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-2678: -- Priority: Major (was: Minor) > `Spark-submit` overrides user application options > ---

[jira] [Created] (SPARK-2678) `Spark-submit` overrides user application options

2014-07-24 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-2678: - Summary: `Spark-submit` overrides user application options Key: SPARK-2678 URL: https://issues.apache.org/jira/browse/SPARK-2678 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1855) Provide memory-and-local-disk RDD checkpointing

2014-07-24 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073784#comment-14073784 ] koert kuipers commented on SPARK-1855: -- i think this makes sense. we have iterative q

[jira] [Updated] (SPARK-2677) BasicBlockFetchIterator#next can wait forever

2014-07-24 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2677: -- Summary: BasicBlockFetchIterator#next can wait forever (was: BasicBlockFetchIterator#next can

[jira] [Created] (SPARK-2677) BasicBlockFetchIterator#next can be wait forever

2014-07-24 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2677: - Summary: BasicBlockFetchIterator#next can be wait forever Key: SPARK-2677 URL: https://issues.apache.org/jira/browse/SPARK-2677 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-2250) show stage RDDs in UI

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2250. Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 1188 [https://

[jira] [Updated] (SPARK-2250) show stage RDDs in UI

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2250: --- Assignee: Neville Li > show stage RDDs in UI > - > > Key:

[jira] [Comment Edited] (SPARK-2387) Remove the stage barrier for better resource utilization

2014-07-24 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073597#comment-14073597 ] Kay Ousterhout edited comment on SPARK-2387 at 7/24/14 8:23 PM:

[jira] [Commented] (SPARK-2387) Remove the stage barrier for better resource utilization

2014-07-24 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073597#comment-14073597 ] Kay Ousterhout commented on SPARK-2387: --- Have you done experiments to understand how

[jira] [Updated] (SPARK-2674) Add date and time types to inferSchema

2014-07-24 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2674: Target Version/s: 1.1.0 > Add date and time types to inferSchema >

[jira] [Assigned] (SPARK-2674) Add date and time types to inferSchema

2014-07-24 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-2674: --- Assignee: Michael Armbrust > Add date and time types to inferSchema > ---

[jira] [Updated] (SPARK-2674) Add date and time types to inferSchema

2014-07-24 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2674: Assignee: Davies Liu (was: Michael Armbrust) > Add date and time types to inferSchema > --

[jira] [Resolved] (SPARK-2037) yarn client mode doesn't support spark.yarn.max.executor.failures

2014-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-2037. -- Resolution: Fixed Fix Version/s: 1.1.0 > yarn client mode doesn't support spark.yarn.max

[jira] [Commented] (SPARK-2671) BlockObjectWriter should create parent directory when the directory doesn't exist

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073548#comment-14073548 ] Apache Spark commented on SPARK-2671: - User 'sarutak' has created a pull request for t

[jira] [Commented] (SPARK-2675) LiveListenerBus should set higher capacity for its event queue

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073546#comment-14073546 ] Apache Spark commented on SPARK-2675: - User 'concretevitamin' has created a pull reque

[jira] [Created] (SPARK-2676) CLONE - LiveListenerBus should set higher capacity for its event queue

2014-07-24 Thread Zongheng Yang (JIRA)
Zongheng Yang created SPARK-2676: Summary: CLONE - LiveListenerBus should set higher capacity for its event queue Key: SPARK-2676 URL: https://issues.apache.org/jira/browse/SPARK-2676 Project: Spark

[jira] [Closed] (SPARK-2676) CLONE - LiveListenerBus should set higher capacity for its event queue

2014-07-24 Thread Zongheng Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zongheng Yang closed SPARK-2676. Resolution: Duplicate > CLONE - LiveListenerBus should set higher capacity for its event queue > -

[jira] [Created] (SPARK-2675) LiveListenerBus should set higher capacity for its event queue

2014-07-24 Thread Zongheng Yang (JIRA)
Zongheng Yang created SPARK-2675: Summary: LiveListenerBus should set higher capacity for its event queue Key: SPARK-2675 URL: https://issues.apache.org/jira/browse/SPARK-2675 Project: Spark

[jira] [Created] (SPARK-2674) Add date and time types to inferSchema

2014-07-24 Thread Hossein Falaki (JIRA)
Hossein Falaki created SPARK-2674: - Summary: Add date and time types to inferSchema Key: SPARK-2674 URL: https://issues.apache.org/jira/browse/SPARK-2674 Project: Spark Issue Type: New Featur

[jira] [Commented] (SPARK-2670) FetchFailedException should be thrown when local fetch has failed

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073539#comment-14073539 ] Apache Spark commented on SPARK-2670: - User 'sarutak' has created a pull request for t

[jira] [Commented] (SPARK-1154) Spark fills up disk with app-* folders

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073521#comment-14073521 ] Andrew Ash commented on SPARK-1154: --- For the record, this is Evan's PR that closed this

[jira] [Commented] (SPARK-2464) Twitter Receiver does not stop correctly when streamingContext.stop is called

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073518#comment-14073518 ] Apache Spark commented on SPARK-2464: - User 'tdas' has created a pull request for this

[jira] [Created] (SPARK-2673) Improve Spark so that we can attach Debugger to Executors easily

2014-07-24 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2673: - Summary: Improve Spark so that we can attach Debugger to Executors easily Key: SPARK-2673 URL: https://issues.apache.org/jira/browse/SPARK-2673 Project: Spark

[jira] [Created] (SPARK-2672) support compressed file in wholeFile()

2014-07-24 Thread Davies Liu (JIRA)
Davies Liu created SPARK-2672: - Summary: support compressed file in wholeFile() Key: SPARK-2672 URL: https://issues.apache.org/jira/browse/SPARK-2672 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-2603) Remove unnecessary toMap and toList in converting Java collections to Scala collections JsonRDD.scala

2014-07-24 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2603. - Resolution: Fixed > Remove unnecessary toMap and toList in converting Java collections to

[jira] [Updated] (SPARK-2603) Remove unnecessary toMap and toList in converting Java collections to Scala collections JsonRDD.scala

2014-07-24 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2603: Fix Version/s: 1.0.2 1.1.0 > Remove unnecessary toMap and toList in conv

[jira] [Created] (SPARK-2671) BlockObjectWriter should create parent directory when the directory doesn't exist

2014-07-24 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2671: - Summary: BlockObjectWriter should create parent directory when the directory doesn't exist Key: SPARK-2671 URL: https://issues.apache.org/jira/browse/SPARK-2671 Pro

[jira] [Resolved] (SPARK-2619) Configurable file-mode for spark/bin folder in the .deb package.

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2619. Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 1531 [https://

[jira] [Updated] (SPARK-2619) Configurable file-mode for spark/bin folder in the .deb package.

2014-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2619: --- Assignee: Christian Tzolov > Configurable file-mode for spark/bin folder in the .deb package.

[jira] [Created] (SPARK-2670) FetchFailedException should be thrown when local fetch has failed

2014-07-24 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2670: - Summary: FetchFailedException should be thrown when local fetch has failed Key: SPARK-2670 URL: https://issues.apache.org/jira/browse/SPARK-2670 Project: Spark

[jira] [Updated] (SPARK-2538) External aggregation in Python

2014-07-24 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-2538: - Priority: Critical (was: Major) > External aggregation in Python > -

[jira] [Commented] (SPARK-2479) Comparing floating-point numbers using relative error in UnitTests

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073430#comment-14073430 ] Apache Spark commented on SPARK-2479: - User 'mengxr' has created a pull request for th

[jira] [Commented] (SPARK-2583) ConnectionManager cannot distinguish whether error occurred or not

2014-07-24 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073419#comment-14073419 ] Kousuke Saruta commented on SPARK-2583: --- I have added some test cases to my PR for t

[jira] [Updated] (SPARK-1264) Documentation for setting heap sizes across all configurations

2014-07-24 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1264: -- Assignee: (was: Aaron Davidson) > Documentation for setting heap sizes across all configura

[jira] [Commented] (SPARK-2668) Support log4j log to yarn container log directory

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073369#comment-14073369 ] Peng Zhang commented on SPARK-2668: --- Yes, this is a common issue for long running tasks

[jira] [Commented] (SPARK-2669) Hadoop configuration is not localised when submitting job in yarn-cluster mode

2014-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073338#comment-14073338 ] Apache Spark commented on SPARK-2669: - User 'redbaron' has created a pull request for

[jira] [Created] (SPARK-2669) Hadoop configuration is not localised when submitting job in yarn-cluster mode

2014-07-24 Thread Maxim Ivanov (JIRA)
Maxim Ivanov created SPARK-2669: --- Summary: Hadoop configuration is not localised when submitting job in yarn-cluster mode Key: SPARK-2669 URL: https://issues.apache.org/jira/browse/SPARK-2669 Project: S

[jira] [Commented] (SPARK-2575) SVMWithSGD throwing Input Validation failed

2014-07-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073329#comment-14073329 ] Xiangrui Meng commented on SPARK-2575: -- [~dbtsai] sent a PR for multinomial logistic

[jira] [Commented] (SPARK-2668) Support log4j log to yarn container log directory

2014-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073310#comment-14073310 ] Thomas Graves commented on SPARK-2668: -- Oh, I see you just want a variable to referen

[jira] [Comment Edited] (SPARK-2668) Support log4j log to yarn container log directory

2014-07-24 Thread Peng Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073278#comment-14073278 ] Peng Zhang edited comment on SPARK-2668 at 7/24/14 3:12 PM: [~

  1   2   >