Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/10780#discussion_r52205244
--- Diff: network/yarn/pom.xml ---
@@ -86,6 +88,15
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8#discussion_r52220649
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10780#issuecomment-180284513
Can do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10780#issuecomment-180338682
I've just pushed out a new version which leaves only jackson and guava as
the relocations: the ones that are most trouble. I could turn off the guava
relocate
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-180374115
rebased against master; switch from scan of HTML view to REST API to
enumerate listings of complete/incomplete apps, add @squito's ? arg redirection
and test
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-179849703
...thx for the feedback. I'm fixing the merge which is now triggering a
regression âmaybe a race condition in test startupâ
apps should go from
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-179850363
..I should add that it depends on the head attempt on the list being
complete; the filter in HistoryServer is very sensitive to ordering. If there's
an incomplete
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/11033#issuecomment-179232563
This now works: I've tested it by creating tokens, saving them to a file.
pointing to in the env var then using spark-submit to bring up a cluster
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/11033
[SPARK-13148] [YARN] [WIP] zero-keytab Oozie application launch
This patch looks for the env var `HADOOP_TOKEN_FILE_LOCATION`, and if set
skips trying to collect new delegation tokens
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-176209826
@squito âhave you had a chance to look at the latest version? I think
I've addressed all your issues, and it is building and testing against the
current master
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-175048241
I'm going to add a warning note here pointing to
[HDFS-5478](https://issues.apache.org/jira/browse/HDFS-5478), some versions of
HDFS aren't picking up changes
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10780#issuecomment-172985031
The latest patch
1. relocates everything except leveldb. Leveldb uses JNI stuff, which
doesn't relocate, and on JVMs with multiple classloaders, can
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10821#issuecomment-172941838
..but if you can't get at it, copy & paste is probably the best way to make
do. Make the new method private[spark] and we could add a test in the history
se
Github user steveloughran closed the pull request at:
https://github.com/apache/spark/pull/10782
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10782#issuecomment-172698197
(closing)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10821#issuecomment-172702504
I've been using `HistoryServer.getAttemptURI()` to do this mapping in the
timeline integration -but that method is `private[history]`. The code here
appears
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10782#issuecomment-172280115
Sean: the PR #10780 is the version against /master; its the one with the
full diff. This is just here to show it works for 1.6 too. How about I close
this one
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/10782#discussion_r49939450
--- Diff: network/yarn/pom.xml ---
@@ -96,6 +107,54
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/10782#discussion_r49939469
--- Diff: network/yarn/pom.xml ---
@@ -86,6 +88,15 @@
+
--- End diff
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/10782
SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters
with Jackson 2.2.3: branch-1.6 patch
This is the patch of PR #10780 applied to branch-1.6, to verify it works
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10780#issuecomment-172157447
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/10780
[SPARK-12807] [YARN] [WIP] Spark External Shuffle not working in Hadoop
clusters with Jackson 2.2.3
Patch to
1. Shade jackson 2.x in spark-yarn-shuffle JAR
2. Use maven failsafe
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10780#issuecomment-172143421
Updated patch which has shading of `com.fasterxml.jackson` ->
`org.spark-project.com.fasterxml.jackson`
There's a test for this working; it uses the
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10648#issuecomment-171525569
I have some opinions too, as with YARN timeline service integration, it's
essentially hooked up to a database, both for publishing and retrieval. It
might
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/8533#issuecomment-170940934
Can I also suggest that the code is "IPv6 ready". Facebook have switched to
IPv6 in their DCs, and
[HADOOP-11890](https://issues.apache.org/jira/bro
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10615#issuecomment-169314710
Is this going to require the new parser JAR on the classpath everywhere, or
will everything excluding CSV parsing still work without it?
---
If your project
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8533#discussion_r48953313
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskLocation.scala
---
@@ -35,16 +37,28 @@ case class ExecutorCacheTaskLocation(override val
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/7753#issuecomment-168497378
One thought here: it'd probably be nice to have a json history with the
new events as part of the history server suite regression tests âthat'll
catch any
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10545#issuecomment-168496890
I should add that i'm thinking of moving the core `rest` package from
`yarn/src/history` to `yarn/src/main`. Why? It adds Hadoop authentication to
Jersey Client
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/10545
[SPARK-1537] [YARN] Add history provider for YARN Application Timeline
Server
This is the successor to PR #5413; it incorporates SPARK-11315 (PR #8744),
which was split out for easier
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168318509
now succeeded by #10545
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168315481
yes, it is still relevant, yes it was awaiting review, no I wasn't
expecting it to be closed
---
If your project is set up for it, you can reply to this email
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168317338
I'm about to resubmit it. The way the code is structured, the 2.6 specific
stuff lives under yarn/src/history, as discussed in earlier points in this PR
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48363721
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364543
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48367142
--- Diff: docs/monitoring.md ---
@@ -69,36 +83,53 @@ follows:
+### Spark configuration options
+
Property
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48368082
--- Diff: docs/monitoring.md ---
@@ -69,36 +83,53 @@ follows:
+### Spark configuration options
+
Property
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364727
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364720
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364439
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -430,8 +517,55 @@ private[history] class FsHistoryProvider
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364502
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48366527
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -21,15 +21,28 @@ import java.net.{HttpURLConnection, URL
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48366891
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -281,6 +296,191 @@ class HistoryServerSuite extends
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48363399
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364066
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -430,8 +517,55 @@ private[history] class FsHistoryProvider
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48366085
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48366097
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r48364810
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-166966798
I've updated the patch with the comments, and reworked how the updated
probe works, removing the need to have provider-specific state cached, returned
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-164740423
yeah, I found it surprisingly complex too ... if I known at the start I'd
have steered clear. And don't worry about any delay; there are some other spark
PRs
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10115#issuecomment-164026158
Has this gone into 1.6.0 or 1.6.1? Because the JIRA is still open and
doesn't say
---
If your project is set up for it, you can reply to this email and have
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9518#discussion_r47209179
--- Diff:
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-163210092
(I may be trusted enough to start a run..let's see)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r47081031
--- Diff: core/pom.xml ---
@@ -40,6 +40,16 @@
${avro.mapred.classifier}
+ com.amazonaws
+ aws-java-sdk
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r47081146
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkS3UtilSuite.scala ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r47081737
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala ---
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9306#issuecomment-163222684
Moving to Hadoop 0.90
[HADOOP-9623](https://issues.apache.org/jira/browse/HADOOP-9623) was what could
be described as "an accidental disaster"';
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-163203560
Failing test is pyspark âpretty unlikely to be related
```
==
FAIL
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r47083416
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -718,6 +719,7 @@ private[spark] object JsonProtocol
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-163209990
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9306#issuecomment-163253041
> However s3a is not a breeze either (even in newer Hadoop 2.7+ versions),
especially with Frankfurt buckets, which support only AWS Signature V4.
rea
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/10227
[SPARK-12241] [YARN] Improve failure reporting in Yarn client
obtainTokenForHBase()
This lines up the HBase token logic with that done for Hive in SPARK-11265:
reflection with only CFNE
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9571#issuecomment-163358148
latest branch cut out health checks. They'd be nice, but as they are useful
across all of spark, it's probably best to wait for something central to go
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9518#discussion_r47142484
--- Diff:
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9518#issuecomment-163373154
(I'm not an admin, so can't verify the patch)
I've just compared it to the Hadoop StatsDSink; it looks pretty similar
although there's some encouragement
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/8744#issuecomment-162963641
Now that it's dependency PR is in, I welcome comments and reviews on this.
The latest change just adds a version counter to every entity publishing, so
that when
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r46991325
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala ---
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-162962369
FWIW, I've got the SPARK-1537 Yarn history provider hooked up to this in [a
branch](https://github.com/steveloughran/spark/tree/history/SPARK-7889%2BSPARK-1537
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r46992376
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala ---
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/8512#issuecomment-162974033
Has anyone looked at the performance of this versus S3a in Hadoop 2.7+?
Because while I do agree this will dramatically improve s3n: and s3: perf, all
ongoing
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-162965678
Jerry: do you know when YARN visits this decisions about having things
pre-emptible? That is: if you are given a warning dos that mean the container
will go any
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46993661
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,648 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46995031
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,648 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46995495
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,579 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47010882
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -281,6 +296,202 @@ class HistoryServerSuite extends
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47012771
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala
---
@@ -73,4 +101,17 @@ private[history] abstract class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47012758
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala
---
@@ -33,7 +33,35 @@ private[spark] case class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47012646
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,648 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47023063
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -281,6 +296,202 @@ class HistoryServerSuite extends
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47017364
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -678,6 +827,54 @@ private[history] class FsHistoryProvider
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47017107
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -430,8 +519,54 @@ private[history] class FsHistoryProvider
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47018390
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -230,6 +230,13 @@ private[spark] class EventLoggingListener
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47018518
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46673276
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -699,12 +853,24 @@ private class FsApplicationAttemptInfo
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46672615
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -430,8 +476,50 @@ private[history] class FsHistoryProvider
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46672772
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala
---
@@ -73,4 +78,34 @@ private[history] abstract class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46672762
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,579 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46673138
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
---
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46673155
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -175,18 +180,34 @@ private[history] class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46672796
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -281,6 +296,199 @@ class HistoryServerSuite extends
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46672830
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ---
@@ -0,0 +1,579 @@
+/*
+ * Licensed to the Apache
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r46700913
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -610,11 +701,38 @@ private[history] class
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-161779559
This is the next iteration; if you look at the intermittent patches I was
trying to track the time the filesize changed, but it (a) made the code complex
(b
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9182#discussion_r46434094
--- Diff:
yarn/src/test/scala/org/apache/spark/scheduler/cluster/ExtensionServiceIntegrationSuite.scala
---
@@ -0,0 +1,87 @@
+/*
+ * Licensed
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9182#discussion_r46432466
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -51,6 +51,64 @@ private[spark] abstract class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9182#discussion_r46432648
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -51,6 +51,64 @@ private[spark] abstract class
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9182#discussion_r46433390
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/SchedulerExtensionService.scala
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-161240157
the reason the modtime doesn't change is the log file is kept open; the mod
time is set when the output stream is created and not updated as new data is
added
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6935#issuecomment-161107339
... now it's looking like somethings up with the modtime info in the local
FS; I'm thinking of tracking the file length as well: a bigger file -> more
reco
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9182#issuecomment-161107529
thanks -will deal with these on wednesday.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
701 - 800 of 1115 matches
Mail list logo
