[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn
runzhiwang commented on a change in pull request #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn URL: https://github.com/apache/incubator-livy/pull/242#discussion_r347186706 ## File path: server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala ## @@ -98,7 +119,37 @@ object SparkYarnApp extends Logging { } } - + private val yarnAppMonitorThread = new Thread() { +override def run() : Unit = { + val executor = Executors.newSingleThreadExecutor + while (true) { +for ((app, appTag) <- appMap) { + Future { +app.monitorLock.synchronized { + val handler = executor.submit(new Runnable { +override def run(): Unit = { + app.monitorSparkYarnApp() +} + }) + try { +// prevent the rpc block of one app from blocking all apps +handler.get(yarnAppMonitorTimeout, MILLISECONDS) + } catch { +case e: Exception => { + handler.cancel(true) + error(s"monitor app: ${appTag} exception:", e) +} + } + if (!app.isRunning) { +appMap -= app + } +} + } +} +Thread.sleep(yarnPollInterval) Review comment: I have updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn
codecov-io edited a comment on issue #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn URL: https://github.com/apache/incubator-livy/pull/242#issuecomment-538712243 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=h1) Report > Merging [#242](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/6c7df77204a5a7bfb04beb9789253120d8d7db6c?src=pr&el=desc) will **decrease** coverage by `0.06%`. > The diff coverage is `67.03%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-livy/pull/242/graphs/tree.svg?width=650&token=0MkVbiUFwE&height=150&src=pr)](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #242 +/- ## - Coverage 67.9% 67.84% -0.07% - Complexity 939 947 +8 Files 102 102 Lines 5876 5921 +45 Branches891 904 +13 + Hits 3990 4017 +27 - Misses 1314 1327 +13 - Partials572 577 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `96.09% <100%> (+0.07%)` | `21 <0> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `64.43% <65.51%> (-1.59%)` | `48 <6> (+8)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `32.87% <0%> (-0.46%)` | `11% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `79.91% <0%> (ø)` | `45% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=footer). Last update [6c7df77...166c2f5](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services