Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-26 Thread Jiang Yan Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/#review186341
---


Ship it!




Ship It!

- Jiang Yan Xu


On Sept. 26, 2017, 3:04 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62230/
> ---
> 
> (Updated Sept. 26, 2017, 3:04 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-7964
> https://issues.apache.org/jira/browse/MESOS-7964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch dispatches `rmdirs` to a single executor instead of multiple
> `AsyncExecutor`s such that heavy-duty pruning events won't occupy all
> worker threads and block other actors.
> 
> 
> Diffs
> -
> 
>   src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
>   src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 
> 
> 
> Diff: https://reviews.apache.org/r/62230/diff/4/
> 
> 
> Testing
> ---
> 
> sudo make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-26 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/
---

(Updated Sept. 26, 2017, 10:04 p.m.)


Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.


Changes
---

Addressed xujyan's comments.


Bugs: MESOS-7964
https://issues.apache.org/jira/browse/MESOS-7964


Repository: mesos


Description
---

This patch dispatches `rmdirs` to a single executor instead of multiple
`AsyncExecutor`s such that heavy-duty pruning events won't occupy all
worker threads and block other actors.


Diffs (updated)
-

  src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
  src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 


Diff: https://reviews.apache.org/r/62230/diff/4/

Changes: https://reviews.apache.org/r/62230/diff/3-4/


Testing
---

sudo make check


Thanks,

Chun-Hung Hsiao

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-26 Thread Jiang Yan Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/#review186298
---


Fix it, then Ship it!





src/slave/gc.cpp
Lines 84 (patched)


We already have

```
  ~Executor()
  {
terminate(process);
wait(process);
  }
```

Does the `stop()` here make a difference?



src/slave/gc.cpp
Line 229 (original), 230 (patched)


There's a `#include ` added for `async` that we can 
remove now.



src/slave/gc_process.hpp
Lines 110 (patched)


Can we add a bit more expanation since the name `executor` is pretty 
generic?

e.g., 

```
// For executing path removals in a separate thread.
```


- Jiang Yan Xu


On Sept. 12, 2017, 1:52 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62230/
> ---
> 
> (Updated Sept. 12, 2017, 1:52 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-7964
> https://issues.apache.org/jira/browse/MESOS-7964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch dispatches `rmdirs` to a single executor instead of multiple
> `AsyncExecutor`s such that heavy-duty pruning events won't occupy all
> worker threads and block other actors.
> 
> 
> Diffs
> -
> 
>   src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
>   src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 
> 
> 
> Diff: https://reviews.apache.org/r/62230/diff/3/
> 
> 
> Testing
> ---
> 
> sudo make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-12 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/#review185263
---



FAIL: Some Mesos tests failed.

Reviews applied: `['62252', '62230']`

Failed command: `C:\mesos\src\mesos-tests.exe --verbose`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/62230

Relevant logs:

- 
[mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/62230/logs/mesos-tests-stdout.log):

```
[   OK ] ContentType/SchedulerTest.SchedulerReconnect/0 (253 ms)
[ RUN  ] ContentType/SchedulerTest.SchedulerReconnect/1
[   OK ] ContentType/SchedulerTest.SchedulerReconnect/1 (241 ms)
[--] 30 tests from ContentType/SchedulerTest (24924 ms total)

[--] 2 tests from ContentTypeAndSSLConfig/SchedulerSSLTest
[ RUN  ] ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/0
[   OK ] ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/0 (961 
ms)
[ RUN  ] ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1
[   OK ] ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1 (995 
ms)
[--] 2 tests from ContentTypeAndSSLConfig/SchedulerSSLTest (2054 ms 
total)

[--] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest
[ RUN  ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0
[   OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0 (131 
ms)
[ RUN  ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1
[   OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1 (160 
ms)
[--] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest (335 ms 
total)

[--] Global test environment tear-down
[==] 627 tests from 66 test cases ran. (333988 ms total)
[  PASSED  ] 625 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] ContentType/MasterAPITest.EventAuthorizationFiltering/1, where 
GetParam() = application/json
[  FAILED  ] MesosContainerizer/DefaultExecutorTest.SigkillExecutor/0, where 
GetParam() = "mesos"

 2 FAILED TESTS
  YOU HAVE 174 DISABLED TESTS

```

- 
[mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/62230/logs/mesos-tests-stderr.log):

```
I0913 05:23:46.984045 14064 master.cpp:8418] Removing framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a- (default)
I0913 05:23:46.985044 14064 master.cpp:3267] Deactivating framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a- (default)
I0913 05:23:46.999840 13252 hierarchical.cpp:412] Deactivated framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a-
I0913 05:23:46.999840 14108 slave.cpp:3235] Shutting down framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a-
I0913 05:23:47.000609 14064 master.cpp:8993] Updating the state of task 
74bc6c1f-13f2-482c-b54f-f99aa688d800 of framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a- (latest state: TASK_KILLED, status 
update state: TASK_KILLED)
I0913 05:23:47.000609 14108 slave.cpp:5731] Shutting down executor 'default' of 
framework 9333fbac-335a-4c17-8fa6-0d297bcc8a1a- (via HTTP)
I0913 05:23:47.004621 14064 master.cpp:9087] Removing task 
74bc6c1f-13f2-482c-b54f-f99aa688d800 with resources 
[{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
 of framework 9333fbac-335a-4c17-8fa6-0d297bcc8a1a- on agent 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a-S0 at slave(254)@10.3.1.5:49240 
(mesos-bld-s1.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net)
I0913 05:23:47.024621 14064 master.cpp:9116] Removing executor 'default' with 
resources [] of framework 9333fbac-335a-4c17-8fa6-0d297bcc8a1a- on agent 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a-S0 at slave(254)@10.3.1.5:49240 
(mesos-bld-s1.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net)
I0913 05:23:47.027621  9232 hierarchical.cpp:355] Removed framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a-
E0913 05:23:47.028898 14108 scheduler.cpp:649] End-Of-File received from 
master. The master closed the event stream
I0913 05:23:47.029620  9232 scheduler.cpp:444] Re-detecting master
I0913 05:23:47.032620 11676 scheduler.cpp:470] New master detected at 
master@10.3.1.5:49240
I0913 05:23:47.047621 11676 slave.cpp:5407] Executor 'default' of framework 
9333fbac-335a-4c17-8fa6-0d297bcc8a1a- exited with status 0
I0913 05:23:47.057621 11676 slave.cpp:5511] Cleaning up executor 'default' of 
framework 9333fbac-335a-4c17-8fa6-0d297bcc8a1a- (via HTTP)
W0913 05:23:47.057621 11832 master.cpp:7021] Ignoring unknown exited executor 
'default' of framework

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-12 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/
---

(Updated Sept. 12, 2017, 8:52 p.m.)


Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.


Changes
---

Used the new `process::Executor::execute()` interface.


Bugs: MESOS-7964
https://issues.apache.org/jira/browse/MESOS-7964


Repository: mesos


Description
---

This patch dispatches `rmdirs` to a single executor instead of multiple
`AsyncExecutor`s such that heavy-duty pruning events won't occupy all
worker threads and block other actors.


Diffs (updated)
-

  src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
  src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 


Diff: https://reviews.apache.org/r/62230/diff/3/

Changes: https://reviews.apache.org/r/62230/diff/2-3/


Testing
---

sudo make test


Thanks,

Chun-Hung Hsiao

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-12 Thread Mesos Reviewbot Windows 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/#review185181
---



FAIL: Mesos failed to build.

Reviews applied: [62230]

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/62230

Relevant logs:

 - mesos-cmake-build.log:

��-- The C compiler identification is 
MSVC 19.10.25019.0

-- The CXX compiler identification is 
MSVC 19.10.25019.0

-- Check for working C compiler: 
C:/Program Files (x86)/Microsoft Visual 
Studio/2017/Community/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/cl.exe

-- Check for working C compiler: 
C:/Program Files (x86)/Microsoft Visual 
Studio/2017/Community/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/cl.exe
 -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Check for working CXX compiler: 
C:/Program Files (x86)/Microsoft Visual 
Studio/2017/Community/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/cl.exe

-- Check for working CXX compiler: 
C:/Program Files (x86)/Microsoft Visual 
Studio/2017/Community/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/cl.exe
 -- works

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - 
done

-- Detecting CXX compile features

-- Detecting CXX compile features - 
done

-- cotire 1.7.10 loaded.

-- 
************************************************************

-- ********* Beginning Mesos CMake 
configuration step *********

-- 
************************************************************

-- INSTALLATION PREFIX: C:/Program 
Files/Mesos

-- MACHINE SPECS:

--     Hostname: 

--     OS:       WINDOWS(10.0.14393)

--     Arch:     AMD64

--     BitMode:  

--     BuildID:  

-- 
************************************************************


Full log available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/62230/logs/mesos-cmake-build.log

- Mesos Reviewbot Windows


On Sept. 12, 2017, 10:37 a.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62230/
> ---
> 
> (Updated Sept. 12, 2017, 10:37 a.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-7964
> https://issues.apache.org/jira/browse/MESOS-7964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch dispatches `rmdirs` to a single executor instead of multiple
> `AsyncExecutor`s such that heavy-duty pruning events won't occupy all
> worker threads and block other actors.
> 
> 
> Diffs
> -
> 
>   src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
>   src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 
> 
> 
> Diff: https://reviews.apache.org/r/62230/diff/2/
> 
> 
> Testing
> ---
> 
> sudo make test
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-11 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/
---

(Updated Sept. 12, 2017, 2:37 a.m.)


Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.


Changes
---

Addressed xujyan's comment.


Bugs: MESOS-7964
https://issues.apache.org/jira/browse/MESOS-7964


Repository: mesos


Description (updated)
---

This patch dispatches `rmdirs` to a single executor instead of multiple
`AsyncExecutor`s such that heavy-duty pruning events won't occupy all
worker threads and block other actors.


Diffs (updated)
-

  src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
  src/slave/gc_process.hpp c383ce28411622692e42401c80e9443e7b1f5099 


Diff: https://reviews.apache.org/r/62230/diff/2/

Changes: https://reviews.apache.org/r/62230/diff/1-2/


Testing
---

sudo make test


Thanks,

Chun-Hung Hsiao

Re: Review Request 62230: Avoid GC pruning events from blocking other processes.

2017-09-11 Thread Jiang Yan Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62230/#review185123
---




src/slave/gc.cpp
Lines 231 (patched)


I was suggesting using this: 
https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/executor.hpp
 as a helper actor with the same lifecycle as the gc actor, and we dispatch all 
rmdirs onto that actor.

We did the previous patch precisely to avoid dispatching onto the gc actor 
itself because the rmdirs could block new task launches.


- Jiang Yan Xu


On Sept. 11, 2017, 2:39 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62230/
> ---
> 
> (Updated Sept. 11, 2017, 2:39 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler, Jie Yu, and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-7964
> https://issues.apache.org/jira/browse/MESOS-7964
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch dispatches GC pruning events to `GarbageCollectorProcess`
> instead of multiple `AsyncExecutor`s such that if multiple heavy-duty
> pruning events won't occupy all worker threads and block other actors.
> 
> 
> Diffs
> -
> 
>   src/slave/gc.cpp 83e4e2f3aba5c0d9900cf0beeea6e92320f889e7 
> 
> 
> Diff: https://reviews.apache.org/r/62230/diff/1/
> 
> 
> Testing
> ---
> 
> sudo make test
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>