[GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43189656 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala --- @@ -142,6 +145,117 @@ class YarnSparkHadoopUtil extends

[GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43189502 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala --- @@ -142,6 +145,117 @@ class YarnSparkHadoopUtil extends

[GitHub] spark pull request: [SPARK-11315] [YARN] WiP Add YARN extension se...

2015-10-27 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-151647359 OK, this is a rework to make the concurrency logic cleaner/clearer; tested against Hadoop 2.6.x, 2.7.x and 2.8.0-SNAPSHOT. The thread which posts events

[GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43128655 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala --- @@ -142,6 +145,117 @@ class YarnSparkHadoopUtil extends

[GitHub] spark pull request: [SPARK-11272][Core][UI] Support importing and ...

2015-10-27 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9238#issuecomment-151462264 1. Adding new abstract methods is going to break providers which use the current plugin point. Adding base implementations which throw an "unsupported"

[GitHub] spark pull request: [SPARK-11272][Core][UI] Support importing and ...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9238#discussion_r43109175 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -299,6 +300,54 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request: [SPARK-11272][Core][UI] Support importing and ...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9238#discussion_r43100632 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala --- @@ -73,4 +73,11 @@ private[history] abstract class

[GitHub] spark pull request: [SPARK-11206] Support SQL UI on the history se...

2015-10-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9297#discussion_r43098850 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -504,9 +542,28 @@ private[spark] object JsonProtocol { case

[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...

2015-10-27 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-151425243 you are right. I'd probably have to do it by releasing an upstream artifact which will wait until run on your jenkins instance before posting your credentia

[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-151291754 ...so if I submit a patch which modifies that package and runs a test which prints your env vars to stdout, I've got your keys. --- If your project is s

[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-151276071 If you use env vars, you'd better make sure they don't ever leak into test reports (e.g no logging of env vars in an executor) --- If your project is

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-151173854 -Updated patch pulls the `val token=` and `fail` code from the tests. It also pulls out the logic for deciding what to do with exceptions into its own class

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-26 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43004016 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-26 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43003603 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-151145666 Thomas: been through the hive versions and they didn't pull anything, simply added another overloaded method before the existing three. Probably a latent b

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-151140385 Thomas, I'm not sure that they actually pulled something, but instead overloaded the get() operation, which confused the reflection logic. --- If your proje

[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...

2015-10-26 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-151103830 Adding some tests here could be an interesting little project. the main thing to do is to have some config file in test/resources, which git is set up to ignore

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-25 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42945056 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-25 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42945007 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-25 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42944993 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala --- @@ -142,6 +145,104 @@ class YarnSparkHadoopUtil extends

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-25 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42944947 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: [SPARK-11265] [YARN] [WIP] YarnClient can't ge...

2015-10-25 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42944952 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,28 @@ class YarnSparkHadoopUtilSuite

[GitHub] spark pull request: SPARK-11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150700891 I've tightened this up and am happy with the code now, except I'm not sure that NoSuchMethodExceptions should be downgraded. It will hide the si

[GitHub] spark pull request: SPARK-1537: pure service API and test service

2015-10-23 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9182#issuecomment-150689362 -Will do. Then I can combine everything nicely. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150688417 Updated patch 1. Moves all arg validation up before any hive class reflection 1. gets all hive class methods before attempting to call any 1. calls

[GitHub] spark pull request: SPARK-11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150658089 Patch with revert posted For reviewers, here some things to consider 1. Which exceptions should trigger log & skip for token, vs which

[GitHub] spark pull request: SPARK-11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42897746 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1337,55 +1337,8 @@ object Client extends Logging { conf

[GitHub] spark pull request: SPARK-11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42897540 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -322,8 +322,10 @@ private[spark] class Client( // multiple times

[GitHub] spark pull request: Stevel/patches/spark 11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150579863 -just pushed up a version with @dougb's suggestion; the check for cluster mode is made before trying to obtain either hive or hbase tokens. Note tha

[GitHub] spark pull request: Stevel/patches/spark 11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42866295 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala --- @@ -337,5 +339,82 @@ object YarnSparkHadoopUtil

[GitHub] spark pull request: Stevel/patches/spark 11265 hive tokens

2015-10-23 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r42866102 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1337,55 +1337,8 @@ object Client extends Logging { conf

[GitHub] spark pull request: Stevel/patches/spark 11265 hive tokens

2015-10-22 Thread steveloughran
GitHub user steveloughran opened a pull request: https://github.com/apache/spark/pull/9232 Stevel/patches/spark 11265 hive tokens This is just the first "it compiles" stage of the patch: it doesn't contain any fixes I've just pulled the current code ou

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-22 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42784907 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: [SPARK-11182] HDFS Delegation Token will be ex...

2015-10-22 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9168#discussion_r42779515 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -130,6 +130,21 @@ class SparkHadoopUtil extends Logging

[GitHub] spark pull request: update twitter4j to 4.0.4 version

2015-10-22 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-150296842 slf4j & log4j are in sync with Spark 1.6. I can't actually see an explicit ref to commons-logging in the spark poms (just lots of exclusions of the ha

[GitHub] spark pull request: update twitter4j to 4.0.4 version

2015-10-22 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9221#issuecomment-150151459 Looking @ the twitter4j poms, their dependencies pull in a com.googlecode.kryo -given the kryo pain its a possible troublespot. However, that com.googlecode

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-21 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42674559 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-21 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42669119 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,765 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-21 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42668737 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-21 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42667047 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: [SPARK-11182] HDFS Delegation Token will be ex...

2015-10-21 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9168#discussion_r42604617 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -130,6 +130,21 @@ class SparkHadoopUtil extends Logging

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42541364 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42540050 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42539302 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42539136 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42539027 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42538300 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42537310 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42536407 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42531200 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42525776 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42525399 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,1048 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r42525188 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnEventListener.scala --- @@ -0,0 +1,45 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-149595658 Test failed as there was a class `StubApplicationReport` which had been deleted but the deletion not pushed up. The YARN `ApplicationReport` class has grown on

[GitHub] spark pull request: SPARK-1537: pure service API and test service

2015-10-20 Thread steveloughran
GitHub user steveloughran opened a pull request: https://github.com/apache/spark/pull/9182 SPARK-1537: pure service API and test service This is purely the yarn/src/main and yarn/src/test bits of the YARN ATS integration: the extension model to load and run implementations of

[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...

2015-10-20 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/5423#issuecomment-149578694 (this is just a rebase + setup for splitting the initial cluster scheduler extension into its own pull-request) --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-20 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-149578878 (this is just a rebase + setup for splitting the initial cluster scheduler extension into its own pull-request) --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-10614 SystemClock uses non-monotonic tim...

2015-10-20 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8766#issuecomment-149554897 If people are happy with that, I'll do that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: [SPARK-11182] HDFS Delegation Token will be ex...

2015-10-20 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9168#issuecomment-149531922 solution 4: fix HDFS. Don't be afraid of filing reports & submitting patches there; it's a better long-term strategy than trying to work around thi

[GitHub] spark pull request: [SPARK-11182] HDFS Delegation Token will be ex...

2015-10-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9168#discussion_r42482462 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -130,6 +132,20 @@ class SparkHadoopUtil extends Logging

[GitHub] spark pull request: [SPARK-11182] HDFS Delegation Token will be ex...

2015-10-19 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9168#discussion_r42392020 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -130,6 +132,20 @@ class SparkHadoopUtil extends Logging

[GitHub] spark pull request: Stevel/patches/spark 8789 sql query suite tabl...

2015-10-19 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8748#issuecomment-14986 will do --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...

2015-10-16 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9144#issuecomment-148662454 It's been in 1.x. In fact, it's been in 0.19 when the {{fs.permission}} one was first tagged as deprecated [fs/permission/AccessControlException.j

[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...

2015-10-13 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/5423#issuecomment-147770129 (this probably isn't going to compile BTW) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-13 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-147749231 I've pushed out a rebase of this against master. 1. The asynchronous publishing of events with retries and, eventually dropped events, appears

[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...

2015-10-13 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/5423#issuecomment-147698799 I've been thinking about this —how about I do the following # have move the history in under yarn/src/history, as discussed above # also as disc

[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...

2015-10-09 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/5423#issuecomment-146959330 Jerry, the code is where it is to isolate it across versions; it started off where it was in the branch 1.2 era, where the split was common, alpha and stable

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-10-06 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-145818164 @tgravescs: ATS v1. What we got from the ATS team was a commitment to not break that API at some ws/v1/ endpoint they'll continue to offer. However,

[GitHub] spark pull request: SPARK-10614 SystemClock uses non-monotonic tim...

2015-09-17 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8766#issuecomment-141160203 Marcelo, I'm not sure I trust nanoTime any more: I did some reading on the topic and don't think code can be confident it works (i.e. is monotonic and

[GitHub] spark pull request: SPARK-10614 SystemClock uses non-monotonic tim...

2015-09-17 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8766#issuecomment-141033106 test failure is `org.apache.spark.streaming.CheckpointSuite.recovery with invertible reduceByKeyAndWindow operation`; appears unrelated. --- If your project is

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-140853826 Wow —thanks for putting in the effort! I've gone through the work and will also review the rest of the code to see if there are any implications for the

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39672695 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala --- @@ -37,9 +38,12 @@ private[spark] class

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39660218 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnExtensionService.scala --- @@ -0,0 +1,198 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39659390 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnExtensionService.scala --- @@ -0,0 +1,198 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39659260 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39658896 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39658764 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39658639 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39657710 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnTimelineUtils.scala --- @@ -0,0 +1,720 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39657485 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39655137 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39654834 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39654447 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39654285 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39654135 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39653693 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39653822 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39653690 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39653626 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39653540 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39645278 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39645121 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39644786 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39643548 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39643240 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39643031 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39638971 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39638893 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39638905 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-1537 publisher-side code and tests

2015-09-16 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8744#discussion_r39638651 --- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala --- @@ -0,0 +1,981 @@ +/* + * Licensed to

<    5   6   7   8   9   10   11   12   >