[
https://issues.apache.org/jira/browse/MESOS-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964674#comment-13964674
]
Ian Downes commented on MESOS-1199:
-----------------------------------
[~vinodkone] It likely can be reduced significantly.
>From looking at the code I think the most expensive operation will be the
>os::process() for each pid when the slave is not the parent. For example,
>[~tknaup] mentioned a slave that had 100 executors running so if the slave was
>restarted then every T ms it would construct a Process object for all 100
>executor pids. This involves reading {noformat}/proc/[pid]/status{noformat}
>and hits the kernel.
A simple test to cat proc status for 200 processes takes ~20 ms and consumes
non-trivial cpu. Listing pids with ps is fast.
{noformat}
$ /usr/bin/time ps xa -o pid > /dev/null
0.00user 0.00system 0:00.00elapsed 88%CPU (0avgtext+0avgdata 3360maxresident)k
0inputs+0outputs (0major+256minor)pagefaults 0swaps
$ ps xa -o pid | tail -n +2 | head -n 200 | /usr/bin/time xargs -I {} cat
/proc/{}/status > /dev/null
0.00user 0.02system 0:00.23elapsed 10%CPU (0avgtext+0avgdata 2592maxresident)k
0inputs+0outputs (0major+41789minor)pagefaults 0swaps
{noformat}
We don't want to be doing this every 10 ms. When/if we do reduce the poll
interval then we need a more efficient test for a process running.
> Subprocess is "slow" -> gated by process::reap poll interval
> ------------------------------------------------------------
>
> Key: MESOS-1199
> URL: https://issues.apache.org/jira/browse/MESOS-1199
> Project: Mesos
> Issue Type: Improvement
> Affects Versions: 0.18.0
> Reporter: Ian Downes
>
> Subprocess uses process::reap to wait on the subprocess pid and set the exit
> status. However, process::reap polls with a one second interval resulting in
> a delay up to the interval duration before the status future is set.
> This means if you need to wait for the subprocess to complete you get hit
> with E(delay) = 0.5 seconds, independent of the execution time. For example,
> the MesosContainerizer uses mesos-fetcher in a Subprocess to fetch the
> executor during launch. At Twitter we fetch a local file, i.e., a very fast
> operation, but the launch is blocked until the mesos-fetcher pid is reaped ->
> adding 0 to 1 seconds for every launch!
> The problem is even worse with a chain of short Subprocesses because after
> the first Subprocess completes you'll be synchronized with the reap interval
> and you'll see nearly the full interval before notification, i.e., 10
> Subprocesses each of << 1 second duration with take ~10 seconds!
> This has become particularly apparent in some new tests I'm working on where
> test durations are now greatly extended with each taking several seconds.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
