[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn

2019-11-17 Thread GitBox 
runzhiwang commented on a change in pull request #242: [LIVY-336] Livy should 
not spawn one thread per job to track the job on Yarn
URL: https://github.com/apache/incubator-livy/pull/242#discussion_r347186706
 
 

 ##
 File path: server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
 ##
 @@ -98,7 +119,37 @@ object SparkYarnApp extends Logging {
 }
   }
 
-
+  private val yarnAppMonitorThread = new Thread() {
+override def run() : Unit = {
+  val executor = Executors.newSingleThreadExecutor
+  while (true) {
+for ((app, appTag) <- appMap) {
+  Future {
+app.monitorLock.synchronized {
+  val handler = executor.submit(new Runnable {
+override def run(): Unit = {
+  app.monitorSparkYarnApp()
+}
+  })
+  try {
+// prevent the rpc block of one app from blocking all apps
+handler.get(yarnAppMonitorTimeout, MILLISECONDS)
+  } catch {
+case e: Exception => {
+  handler.cancel(true)
+  error(s"monitor app: ${appTag} exception:", e)
+}
+  }
+  if (!app.isRunning) {
+appMap -= app
+  }
+}
+  }
+}
+Thread.sleep(yarnPollInterval)
 
 Review comment:
   I have updated


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-livy] codecov-io edited a comment on issue #242: [LIVY-336] Livy should not spawn one thread per job to track the job on Yarn

2019-11-17 Thread GitBox 
codecov-io edited a comment on issue #242: [LIVY-336] Livy should not spawn one 
thread per job to track the job on Yarn
URL: https://github.com/apache/incubator-livy/pull/242#issuecomment-538712243
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=h1) 
Report
   > Merging 
[#242](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-livy/commit/6c7df77204a5a7bfb04beb9789253120d8d7db6c?src=pr&el=desc)
 will **decrease** coverage by `0.06%`.
   > The diff coverage is `67.03%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-livy/pull/242/graphs/tree.svg?width=650&token=0MkVbiUFwE&height=150&src=pr)](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master #242  +/-   ##
   
   - Coverage  67.9%   67.84%   -0.07% 
   - Complexity  939  947   +8 
   
 Files   102  102  
 Lines  5876 5921  +45 
 Branches891  904  +13 
   
   + Hits   3990 4017  +27 
   - Misses 1314 1327  +13 
   - Partials572  577   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==)
 | `96.09% <100%> (+0.07%)` | `21 <0> (ø)` | :arrow_down: |
   | 
[...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=)
 | `64.43% <65.51%> (-1.59%)` | `48 <6> (+8)` | |
   | 
[...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==)
 | `32.87% <0%> (-0.46%)` | `11% <0%> (ø)` | |
   | 
[...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/242/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=)
 | `79.91% <0%> (ø)` | `45% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=footer).
 Last update 
[6c7df77...166c2f5](https://codecov.io/gh/apache/incubator-livy/pull/242?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services