[jira] [Created] (SPARK-3795) Provide scheduler hooks for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3795: -- Summary: Provide scheduler hooks for adding and removing executors Key: SPARK-3795 URL: https://issues.apache.org/jira/browse/SPARK-3795 Project: Spark I

[jira] [Created] (SPARK-3796) Provide shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3796: -- Summary: Provide shuffle service for external block storage Key: SPARK-3796 URL: https://issues.apache.org/jira/browse/SPARK-3796 Project: Spark Issue Ty

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Summary: Add scheduler hooks/heuristics for adding and removing executors (was: Provide sched

[jira] [Updated] (SPARK-3796) Create shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3796: --- Summary: Create shuffle service for external block storage (was: Provide shuffle service for

[jira] [Created] (SPARK-3797) Integrate shuffle service in YARN's pluggable shuffle

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3797: -- Summary: Integrate shuffle service in YARN's pluggable shuffle Key: SPARK-3797 URL: https://issues.apache.org/jira/browse/SPARK-3797 Project: Spark Issue

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need to

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need to

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Component/s: (was: YARN) > Provide elastic scaling within a Spark application > --

[jira] [Updated] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3794: --- Fix Version/s: (was: 1.2.0) > Building spark core fails with specific hadoop version > ---

[jira] [Commented] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159330#comment-14159330 ] Patrick Wendell commented on SPARK-3794: Are you trying to build from within the c

[jira] [Reopened] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3464: > Graceful decommission of executors > -- > > Ke

[jira] [Resolved] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3464. Resolution: Invalid This is going to be subsumed by a separate JIRA. > Graceful decommissio

[jira] [Commented] (SPARK-3805) Enable Standalone worker cleanup by default

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159837#comment-14159837 ] Patrick Wendell commented on SPARK-3805: I commented on the PR - but because this

[jira] [Commented] (SPARK-3561) Native Hadoop/YARN integration for batch/ETL workloads.

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159838#comment-14159838 ] Patrick Wendell commented on SPARK-3561: [~ozhurakousky] do you have a timeline fo

[jira] [Updated] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3561: --- Summary: Allow for pluggable execution contexts in Spark (was: Native Hadoop/YARN integration

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159840#comment-14159840 ] Patrick Wendell commented on SPARK-3561: I also changed the title here that reflec

[jira] [Comment Edited] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159840#comment-14159840 ] Patrick Wendell edited comment on SPARK-3561 at 10/6/14 3:55 AM: ---

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Priority: Major (was: Blocker) > Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Component/s: Spark Core > Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4 >

[jira] [Updated] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3761: --- Priority: Critical (was: Major) > Class anonfun$1 not found exception / sbt 13.x / Scala 2.10

[jira] [Updated] (SPARK-3783) The type parameters for SparkContext.accumulable are inconsistent Accumulable itself

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3783: --- Assignee: Nathan Kronenfeld > The type parameters for SparkContext.accumulable are inconsisten

[jira] [Resolved] (SPARK-3783) The type parameters for SparkContext.accumulable are inconsistent Accumulable itself

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3783. Resolution: Fixed Fix Version/s: 1.2.0 https://github.com/apache/spark/pull/2637 > T

[jira] [Commented] (SPARK-3782) AkkaUtils directly using log4j

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159846#comment-14159846 ] Patrick Wendell commented on SPARK-3782: Would you mind pasting the exact exceptio

[jira] [Updated] (SPARK-3782) Direct use of log4j in AkkaUtils interferes with certain logging configurations

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3782: --- Summary: Direct use of log4j in AkkaUtils interferes with certain logging configurations (wa

[jira] [Resolved] (SPARK-3314) Script creation of AMIs

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3314. Resolution: Fixed Assignee: Patrick Wendell So I think the original goal of this JIRA

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Assignee: (was: Patrick Wendell) > Allow printing object graph of tasks/RDD's with a debug

[jira] [Updated] (SPARK-1190) Do not initialize log4j if slf4j log4j backend is not being used

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1190: --- Assignee: Patrick Wendell (was: Patrick Cogan) > Do not initialize log4j if slf4j log4j backe

[jira] [Updated] (SPARK-3805) Enable Standalone worker cleanup by default

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3805: --- Component/s: Deploy > Enable Standalone worker cleanup by default > --

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Component/s: YARN Spark Core > Provide elastic scaling within a Spark applica

[jira] [Updated] (SPARK-3765) Add test information to sbt build docs

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3765: --- Summary: Add test information to sbt build docs (was: add testing with sbt to doc) > Add tes

[jira] [Commented] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160002#comment-14160002 ] Patrick Wendell commented on SPARK-3731: [~davies] any chance you can take a look

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Priority: Critical (was: Major) > RDD caching stops working in pyspark after some time >

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Target Version/s: 1.1.1, 1.2.0 > RDD caching stops working in pyspark after some time > --

[jira] [Updated] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3731: --- Assignee: Davies Liu > RDD caching stops working in pyspark after some time >

[jira] [Updated] (SPARK-3742) Link to Spark UI sometimes fails when using H/A RM's

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3742: --- Summary: Link to Spark UI sometimes fails when using H/A RM's (was: SparkUI page unable to op

[jira] [Updated] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3732: --- Component/s: YARN > Yarn Client: Add option to NOT System.exit() at end of main() > --

[jira] [Updated] (SPARK-3742) SparkUI page unable to open

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3742: --- Component/s: (was: Spark Core) YARN > SparkUI page unable to open > -

[jira] [Updated] (SPARK-2885) All-pairs similarity via DIMSUM

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2885: --- Component/s: MLlib > All-pairs similarity via DIMSUM > --- > >

[jira] [Updated] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2331: --- Summary: SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T] (was: SparkContext.emptyR

[jira] [Updated] (SPARK-3782) Direct use of log4j in AkkaUtils interferes with certain logging configurations

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3782: --- Component/s: Spark Core > Direct use of log4j in AkkaUtils interferes with certain logging >

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Component/s: Spark Core > Allow printing object graph of tasks/RDD's with a debug flag > -

[jira] [Updated] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2331: --- Component/s: Spark Core > SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T] > ---

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Issue Type: Improvement (was: Bug) > Allow printing object graph of tasks/RDD's with a debug

[jira] [Updated] (SPARK-3461) Support external groupBy

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Assignee: Davies Liu (was: Patrick Wendell) > Support external groupBy >

[jira] [Updated] (SPARK-3461) Support external groupBy and groupByKey using repartitionAndSortWithinPartitions

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Summary: Support external groupBy and groupByKey using repartitionAndSortWithinPartitions (wa

[jira] [Updated] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Summary: Support external groupByKey using repartitionAndSortWithinPartitions (was: Support e

[jira] [Updated] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Description: Given that we have SPARK-2978, it seems like we could support an external group

[jira] [Updated] (SPARK-3461) Support external groupBy and groupByKey using repartitionAndSortWithinPartitions

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Description: Given that we have SPARK-2978, it seems like we could support an external group

[jira] [Updated] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3633: --- Priority: Critical (was: Major) > Fetches failure observed after SPARK-2711 > ---

[jira] [Updated] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3633: --- Target Version/s: 1.2.0 > Fetches failure observed after SPARK-2711 >

[jira] [Created] (SPARK-3824) Spark SQL should cache in MEMORY_AND_DISK by default

2014-10-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3824: -- Summary: Spark SQL should cache in MEMORY_AND_DISK by default Key: SPARK-3824 URL: https://issues.apache.org/jira/browse/SPARK-3824 Project: Spark Issue

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161382#comment-14161382 ] Patrick Wendell commented on SPARK-2585: Hey [~joshrosen] what happens if you run

[jira] [Comment Edited] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161382#comment-14161382 ] Patrick Wendell edited comment on SPARK-2585 at 10/7/14 2:33 AM: ---

[jira] [Updated] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3778: --- Target Version/s: 1.1.1, 1.2.0 (was: 1.1.1) > newAPIHadoopRDD doesn't properly pass credentia

[jira] [Commented] (SPARK-3819) Jenkins should compile Spark against multiple versions of Hadoop

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161513#comment-14161513 ] Patrick Wendell commented on SPARK-3819: It's not feasible to run against multiple

[jira] [Updated] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Priority: Critical (was: Major) > Support external groupByKey using repartitionAndSortWithinP

[jira] [Resolved] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3828. Resolution: Not a Problem I believe this issue is simply due to different behavior in the Ha

[jira] [Commented] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161564#comment-14161564 ] Patrick Wendell commented on SPARK-3828: Yeah fair point - I think for now though

[jira] [Commented] (SPARK-3797) Run the shuffle service inside the YARN NodeManager as an AuxiliaryService

2014-10-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162351#comment-14162351 ] Patrick Wendell commented on SPARK-3797: For the dependencies issue - the plan is

[jira] [Updated] (SPARK-3765) Add test information to sbt build docs

2014-10-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3765: --- Fix Version/s: 1.2.0 > Add test information to sbt build docs > --

[jira] [Resolved] (SPARK-3765) Add test information to sbt build docs

2014-10-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3765. Resolution: Fixed Assignee: wangfei This was resolved by: https://github.com/apache/sp

[jira] [Resolved] (SPARK-2582) Make Block Manager Master pluggable

2014-10-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2582. Resolution: Won't Fix I closed this PR a long time ago - but didn't close the asscicated JIR

[jira] [Resolved] (SPARK-3836) Spark REPL optionally propagate internal exceptions

2014-10-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3836. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Ahir Reddy https://github.c

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164019#comment-14164019 ] Patrick Wendell commented on SPARK-3561: I wanted to wait for a more complete desi

[jira] [Commented] (SPARK-3847) Enum.hashCode is only consistent within the same JVM

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164246#comment-14164246 ] Patrick Wendell commented on SPARK-3847: We might be able to reflect on the key ty

[jira] [Resolved] (SPARK-3843) Cleanup scalastyle.txt at the end of running dev/scalastyle

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3843. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Kousuke Saruta

[jira] [Commented] (SPARK-3637) NPE in ShuffleMapTask

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164286#comment-14164286 ] Patrick Wendell commented on SPARK-3637: Can you provide a small example that repr

[jira] [Updated] (SPARK-3637) NPE in ShuffleMapTask

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3637: --- Component/s: Spark Core > NPE in ShuffleMapTask > - > > Ke

[jira] [Commented] (SPARK-3819) Jenkins should compile Spark against multiple versions of Hadoop

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164453#comment-14164453 ] Patrick Wendell commented on SPARK-3819: Sorry I should have been more clear. We a

[jira] [Commented] (SPARK-3736) Workers should reconnect to Master if disconnected

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164578#comment-14164578 ] Patrick Wendell commented on SPARK-3736: I spoke a bit offline with [~ilikerps] ab

[jira] [Updated] (SPARK-3736) Workers should reconnect to Master if disconnected

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3736: --- Assignee: Matthew Cheah > Workers should reconnect to Master if disconnected > ---

[jira] [Resolved] (SPARK-2805) Update akka to version 2.3.4

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2805. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Anand Avati > Update akka t

[jira] [Updated] (SPARK-2805) Update akka to version 2.3.4

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2805: --- Summary: Update akka to version 2.3.4 (was: Update akka to version 2.3) > Update akka to vers

[jira] [Updated] (SPARK-2805) Update akka to version 2.3

2014-10-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2805: --- Summary: Update akka to version 2.3 (was: update akka to version 2.3) > Update akka to versio

[jira] [Commented] (SPARK-3873) Scala style: check import ordering

2014-10-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164841#comment-14164841 ] Patrick Wendell commented on SPARK-3873: If we can do this it would be super, dupe

[jira] [Created] (SPARK-3874) Provide stable TaskContext API

2014-10-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3874: -- Summary: Provide stable TaskContext API Key: SPARK-3874 URL: https://issues.apache.org/jira/browse/SPARK-3874 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-3883) Provide SSL support for Akka and HttpServer based connections

2014-10-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3883: --- Issue Type: Improvement (was: Story) > Provide SSL support for Akka and HttpServer based conn

[jira] [Reopened] (SPARK-2805) Update akka to version 2.3.4

2014-10-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2805: I had to revert this because of a dependency issue with protobuf. > Update akka to version 2.3.

[jira] [Updated] (SPARK-3892) Map type should have typeName

2014-10-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3892: --- Component/s: SQL > Map type should have typeName > - > >

[jira] [Resolved] (SPARK-2805) Update akka to version 2.3.4

2014-10-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2805. Resolution: Fixed We've merged again with some modifications and we'll see if it works well

[jira] [Updated] (SPARK-3855) Binding Exception when running PythonUDFs

2014-10-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3855: --- Component/s: PySpark > Binding Exception when running PythonUDFs > ---

[jira] [Updated] (SPARK-3855) Binding Exception when running PythonUDFs

2014-10-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3855: --- Component/s: SQL > Binding Exception when running PythonUDFs > ---

[jira] [Resolved] (SPARK-3894) Scala style: line length increase to 160

2014-10-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3894. Resolution: Won't Fix I don't think we are interested in changing this. Feel free to start a

[jira] [Updated] (SPARK-2075) Anonymous classes are missing from Spark distribution

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2075: --- Summary: Anonymous classes are missing from Spark distribution (was: Hadoop1 distribution of

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168727#comment-14168727 ] Patrick Wendell commented on SPARK-2593: Yeah for Spark Streaming the API visibili

[jira] [Commented] (SPARK-3915) backport 'spark.localExecution.enabled' to 1.0

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168947#comment-14168947 ] Patrick Wendell commented on SPARK-3915: This option won't be relevant to 1.0 I th

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168948#comment-14168948 ] Patrick Wendell commented on SPARK-3431: If we can get the maven build times down

[jira] [Comment Edited] (SPARK-3897) Scala style: format example code

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168949#comment-14168949 ] Patrick Wendell edited comment on SPARK-3897 at 10/13/14 5:04 AM: --

[jira] [Commented] (SPARK-3897) Scala style: format example code

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168949#comment-14168949 ] Patrick Wendell commented on SPARK-3897: Hey [~sjk] - can you modify your JIRA acc

[jira] [Comment Edited] (SPARK-3897) Scala style: format example code

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168949#comment-14168949 ] Patrick Wendell edited comment on SPARK-3897 at 10/13/14 5:04 AM: --

[jira] [Commented] (SPARK-3869) ./bin/spark-class miss Java version with _JAVA_OPTIONS set

2014-10-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168951#comment-14168951 ] Patrick Wendell commented on SPARK-3869: Hey [~cocoatomo] would you mind updating

[jira] [Updated] (SPARK-1010) Update all unit tests to use SparkConf instead of system properties

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1010: --- Reporter: Patrick Wendell (was: Patrick Cogan) > Update all unit tests to use SparkConf inste

[jira] [Resolved] (SPARK-2707) Upgrade to Akka 2.3

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2707. Resolution: Fixed This was fixed in SPARK-2805 > Upgrade to Akka 2.3 > ---

[jira] [Updated] (SPARK-3925) Do not consider the ordering of qualifiers during comparison

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3925: --- Component/s: SQL > Do not consider the ordering of qualifiers during comparison >

[jira] [Created] (SPARK-3937) Unsafe memory access inside of Snappy library

2014-10-13 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3937: -- Summary: Unsafe memory access inside of Snappy library Key: SPARK-3937 URL: https://issues.apache.org/jira/browse/SPARK-3937 Project: Spark Issue Type: B

[jira] [Created] (SPARK-3938) Set RDD name to table name during cache operations

2014-10-13 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3938: -- Summary: Set RDD name to table name during cache operations Key: SPARK-3938 URL: https://issues.apache.org/jira/browse/SPARK-3938 Project: Spark Issue Ty

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170298#comment-14170298 ] Patrick Wendell commented on SPARK-1209: It's possible this previously used packag

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170302#comment-14170302 ] Patrick Wendell commented on SPARK-1209: Actually a full set of forwarders might b

[jira] [Resolved] (SPARK-3894) Scala style: line length increase to 120 for standard

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3894. Resolution: Won't Fix As I said before - send an e-mail to dev@ if you want to start a discu

[jira] [Updated] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1209: --- Reporter: Sandy Ryza (was: Sandy Pérez González) > SparkHadoopUtil should not use package org

<    8   9   10   11   12   13   14   15   >