[jira] [Updated] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3561: --- Summary: Allow for pluggable execution contexts in Spark (was: Native Hadoop/YARN

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159840#comment-14159840 ] Patrick Wendell commented on SPARK-3561: I also changed the title here

[jira] [Comment Edited] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159840#comment-14159840 ] Patrick Wendell edited comment on SPARK-3561 at 10/6/14 3:55 AM

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Priority: Major (was: Blocker) Class anonfun$1 not found exception / sbt 13.x / Scala

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Component/s: Spark Core Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Priority: Critical (was: Major) Class anonfun$1 not found exception / sbt 13.x / Scala

[jira] [Updated] (SPARK-3783) The type parameters for SparkContext.accumulable are inconsistent Accumulable itself

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3783: --- Assignee: Nathan Kronenfeld The type parameters for SparkContext.accumulable

[jira] [Resolved] (SPARK-3783) The type parameters for SparkContext.accumulable are inconsistent Accumulable itself

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3783. Resolution: Fixed Fix Version/s: 1.2.0 https://github.com/apache/spark/pull/2637

[jira] [Commented] (SPARK-3782) AkkaUtils directly using log4j

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159846#comment-14159846 ] Patrick Wendell commented on SPARK-3782: Would you mind pasting the exact

[jira] [Updated] (SPARK-3782) Direct use of log4j in AkkaUtils interferes with certain logging configurations

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3782: --- Summary: Direct use of log4j in AkkaUtils interferes with certain logging configurations

[jira] [Resolved] (SPARK-3314) Script creation of AMIs

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3314. Resolution: Fixed Assignee: Patrick Wendell So I think the original goal of this JIRA

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Assignee: (was: Patrick Wendell) Allow printing object graph of tasks/RDD's with a debug

[jira] [Updated] (SPARK-1190) Do not initialize log4j if slf4j log4j backend is not being used

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1190: --- Assignee: Patrick Wendell (was: Patrick Cogan) Do not initialize log4j if slf4j log4j

[jira] [Updated] (SPARK-3805) Enable Standalone worker cleanup by default

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3805: --- Component/s: Deploy Enable Standalone worker cleanup by default

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Component/s: YARN Spark Core Provide elastic scaling within a Spark

[jira] [Updated] (SPARK-3765) Add test information to sbt build docs

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3765: --- Summary: Add test information to sbt build docs (was: add testing with sbt to doc) Add

[jira] [Commented] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14160002#comment-14160002 ] Patrick Wendell commented on SPARK-3731: [~davies] any chance you can take a look

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Priority: Critical (was: Major) RDD caching stops working in pyspark after some time

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Target Version/s: 1.1.1, 1.2.0 RDD caching stops working in pyspark after some time

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Assignee: Davies Liu RDD caching stops working in pyspark after some time

[jira] [Updated] (SPARK-3742) Link to Spark UI sometimes fails when using H/A RM's

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3742: --- Summary: Link to Spark UI sometimes fails when using H/A RM's (was: SparkUI page unable

[jira] [Updated] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3732: --- Component/s: YARN Yarn Client: Add option to NOT System.exit() at end of main

[jira] [Updated] (SPARK-3742) SparkUI page unable to open

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3742: --- Component/s: (was: Spark Core) YARN SparkUI page unable to open

[jira] [Updated] (SPARK-2885) All-pairs similarity via DIMSUM

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2885: --- Component/s: MLlib All-pairs similarity via DIMSUM

[jira] [Updated] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2331: --- Summary: SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T

[jira] [Updated] (SPARK-3782) Direct use of log4j in AkkaUtils interferes with certain logging configurations

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3782: --- Component/s: Spark Core Direct use of log4j in AkkaUtils interferes with certain logging

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Component/s: Spark Core Allow printing object graph of tasks/RDD's with a debug flag

[jira] [Updated] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2331: --- Component/s: Spark Core SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Issue Type: Improvement (was: Bug) Allow printing object graph of tasks/RDD's with a debug

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Summary: Provide elastic scaling within a Spark application (was: Under YARN, add and remove

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159250#comment-14159250 ] Patrick Wendell commented on SPARK-3174: Hey all - I'm going to restructure

[jira] [Updated] (SPARK-3460) Under YARN, discard executors that have been idle

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3460: --- Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-3174) Under YARN, discard

[jira] [Updated] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3464: --- Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-3174) Graceful decommission

[jira] [Resolved] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3464. Resolution: Fixed Graceful decommission of executors

[jira] [Created] (SPARK-3795) Provide scheduler hooks for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3795: -- Summary: Provide scheduler hooks for adding and removing executors Key: SPARK-3795 URL: https://issues.apache.org/jira/browse/SPARK-3795 Project: Spark

[jira] [Created] (SPARK-3796) Provide shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3796: -- Summary: Provide shuffle service for external block storage Key: SPARK-3796 URL: https://issues.apache.org/jira/browse/SPARK-3796 Project: Spark Issue

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Summary: Add scheduler hooks/heuristics for adding and removing executors (was: Provide

[jira] [Updated] (SPARK-3796) Create shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3796: --- Summary: Create shuffle service for external block storage (was: Provide shuffle service

[jira] [Created] (SPARK-3797) Integrate shuffle service in YARN's pluggable shuffle

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3797: -- Summary: Integrate shuffle service in YARN's pluggable shuffle Key: SPARK-3797 URL: https://issues.apache.org/jira/browse/SPARK-3797 Project: Spark

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Component/s: (was: YARN) Provide elastic scaling within a Spark application

[jira] [Updated] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3794: --- Fix Version/s: (was: 1.2.0) Building spark core fails with specific hadoop version

[jira] [Commented] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159330#comment-14159330 ] Patrick Wendell commented on SPARK-3794: Are you trying to build from within

[jira] [Reopened] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3464: Graceful decommission of executors -- Key

[jira] [Resolved] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3464. Resolution: Invalid This is going to be subsumed by a separate JIRA. Graceful

Re: EC2 clusters ready in launch time + 30 seconds

2014-10-04 Thread Patrick Wendell
Hey All, Just a couple notes. I recently posted a shell script for creating the AMI's from a clean Amazon Linux AMI. https://github.com/mesos/spark-ec2/blob/v3/create_image.sh I think I will update the AMI's soon to get the most recent security updates. For spark-ec2's purpose this is probably

[jira] [Resolved] (SPARK-3007) Add Dynamic Partition support to Spark Sql hive

2014-10-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3007. Resolution: Fixed Okay, this was merged again: https://github.com/apache/spark/pull/2616

[jira] [Commented] (SPARK-3573) Dataset

2014-10-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156129#comment-14156129 ] Patrick Wendell commented on SPARK-3573: I think people are hung up on the term

[jira] [Resolved] (SPARK-1767) Prefer HDFS-cached replicas when scheduling data-local tasks

2014-10-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1767. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Colin Patrick McCabe Fixed

[jira] [Resolved] (SPARK-3757) mvn clean doesn't delete some files

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3757. Resolution: Fixed Fix Version/s: 1.2.0 Resolved by: https://github.com/apache/spark

[jira] [Resolved] (SPARK-3756) Include possible MultiException when detecting port collisions

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3756. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee

[jira] [Updated] (SPARK-3756) Include possible MultiException when detecting port collisions

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3756: --- Summary: Include possible MultiException when detecting port collisions (was: check

[jira] [Reopened] (SPARK-3755) Do not bind port 1 - 1024 to server in spark

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3755: Original fix was broken so I'm re-opening it. Do not bind port 1 - 1024 to server in spark

[jira] [Resolved] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3292. Resolution: Won't Fix Seems like this is a necessary feature of the current design and can

[jira] [Commented] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155829#comment-14155829 ] Patrick Wendell commented on SPARK-3761: Is this an sbt bug rather than a spark

[jira] [Comment Edited] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155829#comment-14155829 ] Patrick Wendell edited comment on SPARK-3761 at 10/2/14 12:03 AM

[jira] [Commented] (SPARK-3759) SparkSubmitDriverBootstrapper should return exit code of driver process

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155833#comment-14155833 ] Patrick Wendell commented on SPARK-3759: Thanks for reporting this Eric - do you

[jira] [Resolved] (SPARK-3730) Any one else having building spark recently

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3730. Resolution: Invalid For this type of question checkout the Spark dev list. To answer your

[jira] [Commented] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155844#comment-14155844 ] Patrick Wendell commented on SPARK-3731: Thanks for reporting this - can you

Re: Extending Scala style checks

2014-10-01 Thread Patrick Wendell
Hey Nick, We can always take built-in rules. Back when we added this Prashant Sharma actually did some great work that lets us write our own style rules in cases where rules don't exist. You can see some existing rules here:

[jira] [Reopened] (SPARK-3007) Add Dynamic Partition support to Spark Sql hive

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3007: This was reverted based on causing large numbers of test failures. Add Dynamic Partition

[jira] [Reopened] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2778: This has been reverted due to failing tests. Add unit tests for Yarn integration

[jira] [Updated] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2778: --- Attachment: yarn-logs.txt I'm attaching logs from the bad test. Add unit tests for Yarn

[jira] [Created] (SPARK-3744) FlumeStreamSuite will fail during port contention

2014-09-30 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3744: -- Summary: FlumeStreamSuite will fail during port contention Key: SPARK-3744 URL: https://issues.apache.org/jira/browse/SPARK-3744 Project: Spark Issue

[jira] [Updated] (SPARK-2626) Stop SparkContext in all examples

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2626: --- Labels: starter (was: ) Stop SparkContext in all examples

[jira] [Commented] (SPARK-2805) update akka to version 2.3

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151976#comment-14151976 ] Patrick Wendell commented on SPARK-2805: I'm working on publishing akka today

[jira] [Commented] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151980#comment-14151980 ] Patrick Wendell commented on SPARK-3479: I think this is more than minor

[jira] [Updated] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3479: --- Priority: Major (was: Minor) Have Jenkins show which tests failed in his GitHub messages

[jira] [Updated] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3479: --- Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-2230) Have Jenkins

[jira] [Resolved] (SPARK-2230) Improvements to Jenkins QA Harness

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2230. Resolution: Fixed This was tracking an earlier initiative to clean up this harness. Since

[jira] [Commented] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152018#comment-14152018 ] Patrick Wendell commented on SPARK-2331: Yeah we could have made this a wider type

[jira] [Resolved] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2331. Resolution: Won't Fix SparkContext.emptyRDD has wrong return type

[jira] [Comment Edited] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152018#comment-14152018 ] Patrick Wendell edited comment on SPARK-2331 at 9/29/14 6:34 PM

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Labels: starter (was: ) Allow printing object graph of tasks/RDD's with a debug flag

[jira] [Updated] (SPARK-2548) JavaRecoverableWordCount is missing

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2548: --- Labels: starter (was: ) JavaRecoverableWordCount is missing

[jira] [Updated] (SPARK-3504) KMeans optimization: track distances and unmoved cluster centers across iterations

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3504: --- Summary: KMeans optimization: track distances and unmoved cluster centers across iterations

[jira] [Commented] (SPARK-3504) KMeans optimization: track distances and unmoved cluster centers across iterations

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152730#comment-14152730 ] Patrick Wendell commented on SPARK-3504: I just updated the title to make it more

[jira] [Commented] (SPARK-3685) Spark's local dir scheme is not configurable

2014-09-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151350#comment-14151350 ] Patrick Wendell commented on SPARK-3685: [~andrewor14] changing the use of local

[jira] [Resolved] (SPARK-2655) Change the default logging level to WARN

2014-09-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2655. Resolution: Won't Fix Change the default logging level to WARN

[jira] [Created] (SPARK-3709) BroadcastSuite.Unpersisting rg.apache.spark.broadcast.BroadcastSuite.Unpersisting TorrentBroadcast is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3709: -- Summary: BroadcastSuite.Unpersisting rg.apache.spark.broadcast.BroadcastSuite.Unpersisting TorrentBroadcast is flaky Key: SPARK-3709 URL: https://issues.apache.org/jira

[jira] [Created] (SPARK-3710) YARN integration test is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3710: -- Summary: YARN integration test is flaky Key: SPARK-3710 URL: https://issues.apache.org/jira/browse/SPARK-3710 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-3710) YARN integration test is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3710: --- Description: This has been regularly failing the master build: Example failure: https

[jira] [Resolved] (SPARK-3695) Enable to show host and port in block fetch failure

2014-09-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3695. Resolution: Fixed Fix Version/s: 1.2.0 Enable to show host and port in block fetch

[jira] [Resolved] (SPARK-2778) Add unit tests for Yarn integration

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2778. Resolution: Fixed Fix Version/s: 1.2.0 Fixed by: https://github.com/apache/spark

[jira] [Commented] (SPARK-3687) Spark hang while processing more than 100 sequence files

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147465#comment-14147465 ] Patrick Wendell commented on SPARK-3687: Can you perform a jstack on the executor

[jira] [Resolved] (SPARK-3576) Provide script for creating the Spark AMI from scratch

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3576. Resolution: Fixed This was fixed in spark-ec2 itself Provide script for creating

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3288: --- Assignee: (was: Andrew Or) All fields in TaskMetrics should be private and use getters

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3288: --- Labels: starter (was: ) All fields in TaskMetrics should be private and use getters/setters

[jira] [Created] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-25 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3694: -- Summary: Allow printing object graph of tasks/RDD's with a debug flag Key: SPARK-3694 URL: https://issues.apache.org/jira/browse/SPARK-3694 Project: Spark

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Description: This would be useful for debugging extra references inside of RDD's Here

[jira] [Resolved] (SPARK-3584) sbin/slaves doesn't work when we use password authentication for SSH

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3584. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Kousuke Saruta

[jira] [Resolved] (SPARK-3686) flume.SparkSinkSuite.Success is flaky

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3686. Resolution: Fixed Resolved by: https://github.com/apache/spark/pull/2531

Re: do MIMA checking before all test cases start?

2014-09-25 Thread Patrick Wendell
, 2014 at 12:04 AM, Patrick Wendell wrote: Have you considered running the mima checks locally? We prefer people not use Jenkins for very frequent checks since it takes resources away from other people trying to run tests. On Wed, Sep 24, 2014 at 6:44 PM, Nan Zhu zhunanmcg...@gmail.com

[jira] [Resolved] (SPARK-3659) Set EC2 version to 1.1.0 in master branch

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3659. Resolution: Fixed Fix Version/s: 1.2.0 https://github.com/apache/spark/pull/2510

[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2691: --- Assignee: Timothy Chen (was: Tim Chen) Allow Spark on Mesos to be launched with Docker

[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2691: --- Assignee: Tim Chen (was: Timothy Hunter) Allow Spark on Mesos to be launched with Docker

[jira] [Resolved] (SPARK-3604) unbounded recursion in getNumPartitions triggers stack overflow for large UnionRDD

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3604. Resolution: Not a Problem unbounded recursion in getNumPartitions triggers stack overflow

[jira] [Updated] (SPARK-3681) Failed to serialized ArrayType or MapType after accessing them in Python

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3681: --- Component/s: PySpark Failed to serialized ArrayType or MapType after accessing them

[jira] [Updated] (SPARK-3663) Document SPARK_LOG_DIR and SPARK_PID_DIR

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3663: --- Component/s: Documentation Document SPARK_LOG_DIR and SPARK_PID_DIR

<    15   16   17   18   19   20   21   22   23   24   >