[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action

2014-09-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146640#comment-14146640 ] Andrew Ash commented on SPARK-3466: --- How would you design this feature? I can imagine

[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action

2014-09-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3466: -- Description: Right now, operations like {{collect()}} and {{take()}} can crash the driver with an OOM

[jira] [Commented] (SPARK-889) Bring back DFS broadcast

2014-09-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146804#comment-14146804 ] Andrew Ash commented on SPARK-889: -- [~matei] should we close ticket this as Won't Fix

[jira] [Created] (SPARK-3663) Document SPARK_LOG_DIR and SPARK_PID_DIR

2014-09-23 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3663: - Summary: Document SPARK_LOG_DIR and SPARK_PID_DIR Key: SPARK-3663 URL: https://issues.apache.org/jira/browse/SPARK-3663 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-09-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145686#comment-14145686 ] Andrew Ash commented on SPARK-3633: --- Hi [~nravi] you can also check the limits on an

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-09-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145754#comment-14145754 ] Andrew Ash commented on SPARK-3633: --- You're getting exceptions but they're not fatal to

[jira] [Created] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-09-21 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3630: - Summary: Identify cause of Kryo+Snappy PARSING_ERROR Key: SPARK-3630 URL: https://issues.apache.org/jira/browse/SPARK-3630 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-09-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3630: -- Description: A recent GraphX commit caused non-deterministic exceptions in unit tests so it was

[jira] [Updated] (SPARK-3614) Filter on minimum occurrences of a term in IDF

2014-09-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3614: -- Assignee: RJ Nowling Filter on minimum occurrences of a term in IDF

[jira] [Updated] (SPARK-3577) Add task metric to report spill time

2014-09-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3577: -- Description: The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into {{ExternalSorter}}.

[jira] [Commented] (SPARK-559) Automatically register all classes used in fields of a class with Kryo

2014-09-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14142830#comment-14142830 ] Andrew Ash commented on SPARK-559: -- As of today in master we're using Twitter Chill

[jira] [Created] (SPARK-3631) Add docs for checkpoint usage

2014-09-21 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3631: - Summary: Add docs for checkpoint usage Key: SPARK-3631 URL: https://issues.apache.org/jira/browse/SPARK-3631 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

2014-09-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2984: -- Description: We've seen several stacktraces and threads on the user mailing list where people are

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

2014-09-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135775#comment-14135775 ] Andrew Ash commented on SPARK-2984: --- [~gphil], S3 consistency varies based on the region

[jira] [Commented] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

2014-09-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135781#comment-14135781 ] Andrew Ash commented on SPARK-3528: --- [~nchammas] it does look like S3 also has the

[jira] [Commented] (SPARK-3400) GraphX unit tests fail nondeterministically

2014-09-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135782#comment-14135782 ] Andrew Ash commented on SPARK-3400: --- [~ankurd] is there another ticket to track the root

[jira] [Updated] (SPARK-3526) Docs section on data locality

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3526: -- Summary: Docs section on data locality (was: Section on data locality) Docs section on data locality

[jira] [Commented] (SPARK-3526) Docs section on data locality

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133662#comment-14133662 ] Andrew Ash commented on SPARK-3526: --- Note: reports from users that reading from

[jira] [Created] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

2014-09-15 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3528: - Summary: Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL Key: SPARK-3528 URL: https://issues.apache.org/jira/browse/SPARK-3528 Project: Spark

[jira] [Comment Edited] (SPARK-3526) Docs section on data locality

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133662#comment-14133662 ] Andrew Ash edited comment on SPARK-3526 at 9/15/14 8:14 AM:

[jira] [Updated] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3528: -- Description: Note that reading from {{file:///.../pom.xml}} is called a PROCESS_LOCAL task {noformat}

[jira] [Updated] (SPARK-1895) Run tests on windows

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1895: -- Description: bin\pyspark python\pyspark\rdd.py Sometimes tests complete without error _. Last

[jira] [Updated] (SPARK-1764) EOF reached before Python server acknowledged

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1764: -- Description: I'm getting EOF reached before Python server acknowledged while using PySpark on Mesos.

[jira] [Updated] (SPARK-2586) Lack of information to figure out connection to Tachyon master is inactive/ down

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2586: -- Description: When you running Spark with Tachyon, when the connection to Tachyon master is down (due

[jira] [Updated] (SPARK-2586) Lack of information to figure out connection to Tachyon master is inactive/ down

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2586: -- Description: When you running Spark with Tachyon, when the connection to Tachyon master is down (due

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134160#comment-14134160 ] Andrew Ash commented on SPARK-1239: --- For large statuses, would we expect that to exceed

[jira] [Commented] (SPARK-3535) Spark on Mesos not correctly setting heap overhead

2014-09-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134592#comment-14134592 ] Andrew Ash commented on SPARK-3535: --- Why does the task need extra memory if the heap

[jira] [Updated] (SPARK-2532) Fix issues with consolidated shuffle

2014-09-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2532: -- Component/s: Shuffle Fix issues with consolidated shuffle

[jira] [Commented] (SPARK-3172) Distinguish between shuffle spill on the map and reduce side

2014-09-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130278#comment-14130278 ] Andrew Ash commented on SPARK-3172: --- Sandy do you mean distinguish between these two in

[jira] [Updated] (SPARK-2791) Fix committing, reverting and state tracking in shuffle file consolidation

2014-09-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2791: -- Component/s: Shuffle Fix committing, reverting and state tracking in shuffle file consolidation

[jira] [Updated] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-09-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1239: -- Component/s: Shuffle Don't fetch all map output statuses at each reducer during shuffles

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-09-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-3277: -- Component/s: Shuffle LZ4 compression cause the the ExternalSort exception

[jira] [Commented] (SPARK-3211) .take() is OOM-prone when there are empty partitions

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124794#comment-14124794 ] Andrew Ash commented on SPARK-3211: --- This was merged into branch-1.1 and develop

[jira] [Created] (SPARK-3429) Don't include the empty string as a defaultAclUser

2014-09-07 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3429: - Summary: Don't include the empty string as a defaultAclUser Key: SPARK-3429 URL: https://issues.apache.org/jira/browse/SPARK-3429 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2004) QA Automation

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2004: -- Component/s: Project Infra QA Automation - Key: SPARK-2004

[jira] [Commented] (SPARK-1667) Jobs never finish successfully once bucket file missing occurred

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124833#comment-14124833 ] Andrew Ash commented on SPARK-1667: --- Hi [~sarutak] it looks like you sent in a better

[jira] [Commented] (SPARK-2858) Default log4j configuration no longer seems to work

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124838#comment-14124838 ] Andrew Ash commented on SPARK-2858: --- Josh mentions in that ticket that the Spark EC2 AMI

[jira] [Updated] (SPARK-2553) CoGroupedRDD unnecessarily allocates a Tuple2 per dep per key

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2553: -- Fix Version/s: 1.1.0 CoGroupedRDD unnecessarily allocates a Tuple2 per dep per key

[jira] [Updated] (SPARK-2574) Avoid allocating new ArrayBuffer in groupByKey's mergeCombiner

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2574: -- Fix Version/s: 1.1.0 Avoid allocating new ArrayBuffer in groupByKey's mergeCombiner

[jira] [Commented] (SPARK-2048) Optimizations to CPU usage of external spilling code

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124841#comment-14124841 ] Andrew Ash commented on SPARK-2048: --- All subtasks of this umbrella task have been

[jira] [Updated] (SPARK-2048) Optimizations to CPU usage of external spilling code

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2048: -- Fix Version/s: 1.1.0 Optimizations to CPU usage of external spilling code

[jira] [Commented] (SPARK-2122) Move aggregation into shuffle implementation

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124842#comment-14124842 ] Andrew Ash commented on SPARK-2122: --- [~jerryshao] is this a dupe of SPARK-2124 ? It

[jira] [Commented] (SPARK-1956) Enable shuffle consolidation by default

2014-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124844#comment-14124844 ] Andrew Ash commented on SPARK-1956: --- [~mridulm80] there has been a significant amount of

[jira] [Created] (SPARK-3428) TaskMetrics for running tasks is missing GC time metrics

2014-09-06 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3428: - Summary: TaskMetrics for running tasks is missing GC time metrics Key: SPARK-3428 URL: https://issues.apache.org/jira/browse/SPARK-3428 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2099) Report TaskMetrics for running tasks

2014-09-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124777#comment-14124777 ] Andrew Ash commented on SPARK-2099: --- Added a followon ticket for GC times as SPARK-3428

[jira] [Commented] (SPARK-1823) ExternalAppendOnlyMap can still OOM if one key is very large

2014-09-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123213#comment-14123213 ] Andrew Ash commented on SPARK-1823: --- // This was not fixed in Spark 1.1 and should be

[jira] [Commented] (SPARK-3280) Made sort-based shuffle the default implementation

2014-09-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123390#comment-14123390 ] Andrew Ash commented on SPARK-3280: --- [~joshrosen] do you have a theory for the cause of

[jira] [Commented] (SPARK-2099) Report TaskMetrics for running tasks

2014-09-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123584#comment-14123584 ] Andrew Ash commented on SPARK-2099: --- I just gave this a runthrough and most of the

[jira] [Created] (SPARK-3211) .take() is OOM-prone when there are empty partitions

2014-08-25 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-3211: - Summary: .take() is OOM-prone when there are empty partitions Key: SPARK-3211 URL: https://issues.apache.org/jira/browse/SPARK-3211 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3045) Make Serializer interface Java friendly

2014-08-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101419#comment-14101419 ] Andrew Ash commented on SPARK-3045: --- This issue can be closed since the PR was merged

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-08-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099251#comment-14099251 ] Andrew Ash commented on SPARK-2546: --- Ok I'll stay on the lookout for this bug and ping

[jira] [Created] (SPARK-2984) FileNotFoundException on _temporary directory

2014-08-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2984: - Summary: FileNotFoundException on _temporary directory Key: SPARK-2984 URL: https://issues.apache.org/jira/browse/SPARK-2984 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

2014-08-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2984: -- Description: We've seen several stacktraces and threads on the user mailing list where people are

[jira] [Commented] (SPARK-2157) Can't write tight firewall rules for Spark

2014-08-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087372#comment-14087372 ] Andrew Ash commented on SPARK-2157: --- [~epahomov] does this look good? [~pwendell] I

[jira] [Commented] (SPARK-2620) case class cannot be used as key for reduce

2014-07-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074195#comment-14074195 ] Andrew Ash commented on SPARK-2620: --- I attempted to write a unit test to demonstrate

[jira] [Commented] (SPARK-1154) Spark fills up disk with app-* folders

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073521#comment-14073521 ] Andrew Ash commented on SPARK-1154: --- For the record, this is Evan's PR that closed this

[jira] [Commented] (SPARK-1044) Default spark logs location in EC2 AMI leads to out-of-disk space pretty soon

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073930#comment-14073930 ] Andrew Ash commented on SPARK-1044: --- Filling up the work dir could be alleviated by

[jira] [Commented] (SPARK-786) Clean up old work directories in standalone worker

2014-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073932#comment-14073932 ] Andrew Ash commented on SPARK-786: -- Agreed. With SPARK-1860 we could re-enable that the

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-07-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064625#comment-14064625 ] Andrew Ash commented on SPARK-2546: --- On the thread: Me: {quote} Reynold's recent

[jira] [Commented] (SPARK-2521) Broadcast RDD object once per TaskSet (instead of sending it for every task)

2014-07-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064626#comment-14064626 ] Andrew Ash commented on SPARK-2521: --- Reynold's PR:

[jira] [Issue Comment Deleted] (SPARK-2521) Broadcast RDD object once per TaskSet (instead of sending it for every task)

2014-07-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2521: -- Comment: was deleted (was: Reynold's PR: https://github.com/apache/spark/pull/1452) Broadcast RDD

[jira] [Created] (SPARK-2546) Configuration object thread safety issue

2014-07-16 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2546: - Summary: Configuration object thread safety issue Key: SPARK-2546 URL: https://issues.apache.org/jira/browse/SPARK-2546 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2292) NullPointerException in JavaPairRDD.mapToPair

2014-06-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2292: -- Attachment: SPARK-2292-aash-repro.tar.gz I was able to repro the exception below: {noformat} 14/06/27

[jira] [Commented] (SPARK-2292) NullPointerException in JavaPairRDD.mapToPair

2014-06-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047027#comment-14047027 ] Andrew Ash commented on SPARK-2292: --- The situation I was using to repro the exception

[jira] [Resolved] (SPARK-1846) RAT checks should exclude logs/ directory

2014-06-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash resolved SPARK-1846. --- Resolution: Fixed Fix Version/s: 1.0.0 RAT checks should exclude logs/ directory

[jira] [Commented] (SPARK-1846) RAT checks should exclude logs/ directory

2014-06-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14040983#comment-14040983 ] Andrew Ash commented on SPARK-1846: --- Yep this was merged in commit

[jira] [Commented] (SPARK-1882) Support dynamic memory sharing in Mesos

2014-06-20 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038987#comment-14038987 ] Andrew Ash commented on SPARK-1882: --- Yeah, for homogeneous environments, I think you can

[jira] [Commented] (SPARK-1949) Servlet 2.5 vs 3.0 conflict in SBT build

2014-06-19 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038441#comment-14038441 ] Andrew Ash commented on SPARK-1949: --- Sean's PR: https://github.com/apache/spark/pull/906

[jira] [Created] (SPARK-2182) Scalastyle rule for unicode operators

2014-06-18 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2182: - Summary: Scalastyle rule for unicode operators Key: SPARK-2182 URL: https://issues.apache.org/jira/browse/SPARK-2182 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-06-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2182: -- Summary: Scalastyle rule blocking unicode operators (was: Scalastyle rule for unicode operators)

[jira] [Commented] (SPARK-2157) Can't write tight firewall rules for Spark

2014-06-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034135#comment-14034135 ] Andrew Ash commented on SPARK-2157: --- I pulled together Egor's work for HttpBroadcast and

[jira] [Resolved] (SPARK-1903) Document Spark's network connections

2014-06-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash resolved SPARK-1903. --- Resolution: Fixed Fix Version/s: 1.0.0 Merged into master and branch-1.0 in time for the

[jira] [Updated] (SPARK-2148) Document custom class as key needing equals() AND hashcode()

2014-06-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2148: -- Description: Several support requests on user@ have been tracked down to using a custom class as the

[jira] [Commented] (SPARK-2148) Document custom class as key needing equals() AND hashcode()

2014-06-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032024#comment-14032024 ] Andrew Ash commented on SPARK-2148: --- https://github.com/apache/spark/pull/1092

[jira] [Created] (SPARK-2077) Log serializer in use on application startup

2014-06-09 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2077: - Summary: Log serializer in use on application startup Key: SPARK-2077 URL: https://issues.apache.org/jira/browse/SPARK-2077 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2077) Log serializer in use on application startup

2014-06-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021670#comment-14021670 ] Andrew Ash commented on SPARK-2077: --- https://github.com/apache/spark/pull/1017 Log

[jira] [Commented] (SPARK-2078) Use ISO8601 date formats in logging

2014-06-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021709#comment-14021709 ] Andrew Ash commented on SPARK-2078: --- https://github.com/apache/spark/pull/1018 Use

[jira] [Created] (SPARK-2078) Use ISO8601 date formats in logging

2014-06-09 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2078: - Summary: Use ISO8601 date formats in logging Key: SPARK-2078 URL: https://issues.apache.org/jira/browse/SPARK-2078 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1944) Document --verbose in spark-shell -h

2014-06-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021726#comment-14021726 ] Andrew Ash commented on SPARK-1944: --- https://github.com/apache/spark/pull/1020

[jira] [Commented] (SPARK-1103) Garbage collect RDD information inside of Spark

2014-05-27 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010099#comment-14010099 ] Andrew Ash commented on SPARK-1103: --- https://github.com/apache/spark/pull/126 Garbage

[jira] [Updated] (SPARK-1944) Document --verbose in spark-shell -h

2014-05-27 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1944: -- Summary: Document --verbose in spark-shell -h (was: Document --verbse in spark-shell -h) Document

[jira] [Created] (SPARK-1944) Document --verbse in spark-shell -h

2014-05-27 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1944: - Summary: Document --verbse in spark-shell -h Key: SPARK-1944 URL: https://issues.apache.org/jira/browse/SPARK-1944 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008440#comment-14008440 ] Andrew Ash commented on SPARK-983: -- Because ExternalAppendOnlyMap uses

[jira] [Commented] (SPARK-1912) Compression memory issue during reduce

2014-05-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007229#comment-14007229 ] Andrew Ash commented on SPARK-1912: --- https://github.com/apache/spark/pull/860

[jira] [Commented] (SPARK-983) External hashing sorting support

2014-05-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006221#comment-14006221 ] Andrew Ash commented on SPARK-983: -- I think the main intention is to make the .sortByKey()

[jira] [Created] (SPARK-1902) Spark shell prints error when :4040 port already in use

2014-05-22 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1902: - Summary: Spark shell prints error when :4040 port already in use Key: SPARK-1902 URL: https://issues.apache.org/jira/browse/SPARK-1902 Project: Spark Issue Type:

[jira] [Updated] (SPARK-1902) Spark shell prints error when :4040 port already in use

2014-05-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1902: -- Description: When running two shells on the same machine, I get the below error. The issue is that

[jira] [Created] (SPARK-1903) Document Spark's network connections

2014-05-22 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1903: - Summary: Document Spark's network connections Key: SPARK-1903 URL: https://issues.apache.org/jira/browse/SPARK-1903 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running applications

2014-05-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005596#comment-14005596 ] Andrew Ash commented on SPARK-1860: --- So the Spark master webui shows the running

[jira] [Created] (SPARK-1899) Default log4j.properties incorrectly sends all output to stderr and none to stdout

2014-05-21 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1899: - Summary: Default log4j.properties incorrectly sends all output to stderr and none to stdout Key: SPARK-1899 URL: https://issues.apache.org/jira/browse/SPARK-1899 Project:

[jira] [Created] (SPARK-1882) Support dynamic memory sharing in Mesos

2014-05-19 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1882: - Summary: Support dynamic memory sharing in Mesos Key: SPARK-1882 URL: https://issues.apache.org/jira/browse/SPARK-1882 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running applications

2014-05-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001151#comment-14001151 ] Andrew Ash commented on SPARK-1860: --- [~mkim] is going to take a look at this after

[jira] [Created] (SPARK-1846) RAT checks should exclude logs/ directory

2014-05-16 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1846: - Summary: RAT checks should exclude logs/ directory Key: SPARK-1846 URL: https://issues.apache.org/jira/browse/SPARK-1846 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-1757) Support saving null primitives with .saveAsParquetFile()

2014-05-15 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1757: - Summary: Support saving null primitives with .saveAsParquetFile() Key: SPARK-1757 URL: https://issues.apache.org/jira/browse/SPARK-1757 Project: Spark Issue Type:

[jira] [Closed] (SPARK-1757) Support saving null primitives with .saveAsParquetFile()

2014-05-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-1757. - Support saving null primitives with .saveAsParquetFile()

[jira] [Resolved] (SPARK-1829) Sub-second durations shouldn't round to 0 s

2014-05-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash resolved SPARK-1829. --- Resolution: Fixed Didn't make it into v1.0.0-rc6 but is on branch-1.0 so a future rc would likely

[jira] [Commented] (SPARK-1818) Freshen Mesos docs

2014-05-13 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996146#comment-13996146 ] Andrew Ash commented on SPARK-1818: --- https://github.com/apache/spark/pull/756 Freshen

[jira] [Created] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

2014-05-13 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1809: - Summary: Mesos backend doesn't respect HADOOP_CONF_DIR Key: SPARK-1809 URL: https://issues.apache.org/jira/browse/SPARK-1809 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-1757) Support saving null primitives with .saveAsParquetFile()

2014-05-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash resolved SPARK-1757. --- Resolution: Fixed Fix Version/s: 1.0.0 https://github.com/apache/spark/pull/690 Support

[jira] [Commented] (SPARK-1765) Modify a typo in monitoring.md

2014-05-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996012#comment-13996012 ] Andrew Ash commented on SPARK-1765: --- https://github.com/apache/spark/pull/698 This can

<    1   2   3   4   >