Hey,
This code is from std.process:
if (!(config & Config.inheritFDs))
{
import core.sys.posix.sys.resource;
rlimit r;
getrlimit(RLIMIT_NOFILE, &r);
foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
}
This is a close-loop to make sure no fds are leaked before
execve'ing, and can be disabled using Config.inheritFDs.
Here's a program I used to profile:
void main() {
StopWatch sw;
import core.sys.posix.sys.resource;
rlimit r;
errnoEnforce(getrlimit(RLIMIT_NOFILE, &r) == 0);
writefln("The rlimit is %d (hard: %s)", r.rlim_cur,
r.rlim_max);
//r.rlim_cur = 1024 * 1024;
//errnoEnforce(setrlimit(RLIMIT_NOFILE, &r) == 0);
sw.start();
foreach (_; 0 .. 100) {
executeShell("echo hello", null,
std.process.Config.inheritFDs);
}
sw.stop();
writefln("Total time: %s", sw.peek().to!Duration.to!string);
}
These are the results I got:
// On a typical linux machine, without inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 3 secs, 875 ms, 759 μs, and 7 hnsecs
// with inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 80 ms, 549 μs, and 2 hnsecs
// osx, without inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 476 ms, 483 μs, and 5 hnsecs
// with inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 352 ms, 637 μs, and 7 hnsecs
So obviously this becomes a problem if the rlimit is high.
A few suggestions:
1. Make inheritFDs the default, since most people aren't aware of
the performance cost. Also most programs will run just fine if a
few fds are leaked.
1.1. Or, by default, close up to min(rlim_cur, 1024) - this
should be enough for most processes. (If I'm not mistaken, that's
what they do in python). Also this change won't break existing
code.
2. All phobos fds should be opened with O_CLOEXEC, as intended.
This eliminates the need to close all the fds when execve'ing.
3. Optimization for linux - use /proc to check which fds are
actually opened, and close them, instead of making thousands of
syscalls!
