Re: std.process - POSIX specific callback
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote: On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad wrote: On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote: I am aware that std.process is generalized but I doubt such useful functionality which is usable on various Posixen is more disturbing than Windows-only suprpressConsole https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954 I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.) First, suppressConsole is a simple flag, basically passed through to the CreateProcess function, so even though it's Windows-specific, so is the behavior we are suppressing. Consider that when you specify suppressConsole on Posix, the flag works! No console window is created :) But what to do with arbitrary "run this code between fork and exec" on windows? It's not possible. It doesn't belong in the generalized API. But I was mistaken. Config is an enum not struct, so yeah, not worth changing it only for sake of posix callback. So maybe module level variable? module std.process; // ... void delegate() posixPostFork = null; // ... Global state? Don't want to go there... I agree that the global state is a bad idea, ideally you want to specify PER CALL what happens on a fork/exec, not PER THREAD (or PER PROCESS). But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows) I had no time yet to check out the pthread_atfork approach but I see you found some issue with that. posixProcessImpl.spawnProcess and windowsProcessImpl.spawnProcess sounds very good. Then if someone wants to actually take advantage of OS-specific features, they can call on the appropriate object. It shouldn't compile where it's not implemented (e.g. windows spawnProcess shouldn't be callable on Linux). That would be great. Being able to use the easy, simple functions to get the work done and encourage to use them (by Docs and examples) but also allow access to more specialized, "less visible" functions as you proposed. Does this make sense? I think it can be done without breaking any code. May be a lot of boilerplate :) -Steve Thank you very much for looking into this.
Re: std.process - POSIX specific callback
On Tue, 11 Jun 2013 12:31:19 -0400, Lars T. Kyllingstad wrote: On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote: What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows) Then if someone wants to actually take advantage of OS-specific features, they can call on the appropriate object. It shouldn't compile where it's not implemented (e.g. windows spawnProcess shouldn't be callable on Linux). Why should we add an object? Why not expose the OS-specific spawnProcess() implementation as it is -- a free function -- with an additional parameter which specifies a callback delegate? We could do that too. The only thing is that the implementation function is more rough -- it doesn't have all the nice overloads. I was thinking of simply moving the overloads to an object, and then calling on the object's overloads. But thinking about it now, it doesn't make sense. The object's overloads would all have to support this callback parameter. It's probably best to simply expose the underlying implementation. The documentation should explicitly warn about this... From reading the pthread_atfork man page, it is clear that there are very significant problems that can arise from running arbitrary code at that time. Can you think of any other places we'd want to hook besides post-fork-pre-exec? I can't think of anything else at the moment. If we ever support async events in Phobos, we should add an event for "child exited". -Steve
Re: std.process - POSIX specific callback
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote: What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows) Then if someone wants to actually take advantage of OS-specific features, they can call on the appropriate object. It shouldn't compile where it's not implemented (e.g. windows spawnProcess shouldn't be callable on Linux). Why should we add an object? Why not expose the OS-specific spawnProcess() implementation as it is -- a free function -- with an additional parameter which specifies a callback delegate? Can you think of any other places we'd want to hook besides post-fork-pre-exec?
Re: std.process - POSIX specific callback
On Monday, 10 June 2013 at 20:26:59 UTC, Steven Schveighoffer wrote: This is not a good solution. [...] Ok, you make many good points that I hadn't even thought about. I admit, I didn't look too hard at the phtread_atfork() documentation after I'd decided it was just the thing. ;) So maybe we have to do something about this after all. I'll comment on your earlier post.
Re: std.process - POSIX specific callback
On Mon, 10 Jun 2013 16:05:15 -0400, Lars T. Kyllingstad wrote: On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote: But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. But with the pthread_atfork() solution you don't have to. Call that function before you call spawnProcess() or any of the other std.process functions, and it should Just Work (TM). The nice thing here is that it is already part of the POSIX standard, and thus should be available on all relevant systems, and we don't have to adapt our cross-platform API at all. This is not a good solution. It deals with the idea that when forking, only the calling thread is alive, all other threads are dead, and one of those dead threads may hold a lock. Also note that the function pointers are function pointers, not delegates. The idea is that prior to fork, you lock all mutexes you want to be unlocked. Then after fork is called, you unlock those mutexes (thus ensuring no dead threads hold the locks). I don't think it makes for a very good generalized solution to "I want to run this arbitrary code". Also, according to SO, it doesn't even do what it means to do, since the newly created process thread can't unlock the mutexes: http://stackoverflow.com/questions/2620313/how-to-use-pthread-atfork-and-pthread-once-to-reinitialize-mutexes-in-child Not only that, but it seems to be permanent -- there is no "unregister pthread_atfork" call. So this has to be a one-time *process-wide* and permanent solution. If you wanted to run code for this specific call to spawnProcess, and not others, then you are SOL. And finally, if your ultimate purpose is to call exec right after fork (as it is in the general case), you are penalized by having to wait for some mutex to be unlocked in order to fork. -Steve
Re: std.process - POSIX specific callback
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote: But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. But with the pthread_atfork() solution you don't have to. Call that function before you call spawnProcess() or any of the other std.process functions, and it should Just Work (TM). The nice thing here is that it is already part of the POSIX standard, and thus should be available on all relevant systems, and we don't have to adapt our cross-platform API at all.
Re: std.process - POSIX specific callback
On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad wrote: On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote: I am aware that std.process is generalized but I doubt such useful functionality which is usable on various Posixen is more disturbing than Windows-only suprpressConsole https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954 I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.) First, suppressConsole is a simple flag, basically passed through to the CreateProcess function, so even though it's Windows-specific, so is the behavior we are suppressing. Consider that when you specify suppressConsole on Posix, the flag works! No console window is created :) But what to do with arbitrary "run this code between fork and exec" on windows? It's not possible. It doesn't belong in the generalized API. But I was mistaken. Config is an enum not struct, so yeah, not worth changing it only for sake of posix callback. So maybe module level variable? module std.process; // ... void delegate() posixPostFork = null; // ... Global state? Don't want to go there... I agree that the global state is a bad idea, ideally you want to specify PER CALL what happens on a fork/exec, not PER THREAD (or PER PROCESS). But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows) Then if someone wants to actually take advantage of OS-specific features, they can call on the appropriate object. It shouldn't compile where it's not implemented (e.g. windows spawnProcess shouldn't be callable on Linux). Does this make sense? I think it can be done without breaking any code. May be a lot of boilerplate :) -Steve
Re: std.process - POSIX specific callback
On Friday, 7 June 2013 at 08:21:32 UTC, Lars T. Kyllingstad wrote: On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote: Again, I am not forcing anything on you or Steven. I am asking you for opinion on something I find useful because you are the experts and I really like your work on new std.process It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess(). Yeah, now I noticed Dennis response. It may be the solution to my issue yes. Lovely POSIX, you can always count on it *G* Thanks Dennis, Lars for help. Sorry for the noise I caused.
Re: std.process - POSIX specific callback
On Friday, 7 June 2013 at 06:27:32 UTC, dennis luehring wrote: are you talking about http://linux.die.net/man/3/pthread_atfork funktionality? Very interesting. That may be exactly what I need. I will try this out and see how does it play with std.process. Thanks a lot!
Re: std.process - POSIX specific callback
Am 07.06.2013 10:21, schrieb Lars T. Kyllingstad: On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote: Again, I am not forcing anything on you or Steven. I am asking you for opinion on something I find useful because you are the experts and I really like your work on new std.process It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess(). maybe http://sourceware.org/pthreads-win32/ or this nice fork https://github.com/GerHobbelt/pthread-win32 can give a hint how pthread_atfork can "work" under windosw and show a way how to unifie the idea for both worlds
Re: std.process - POSIX specific callback
On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote: Again, I am not forcing anything on you or Steven. I am asking you for opinion on something I find useful because you are the experts and I really like your work on new std.process It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess().
Re: std.process - POSIX specific callback
On Friday, 7 June 2013 at 05:59:24 UTC, Lars T. Kyllingstad wrote: On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote: I am aware that std.process is generalized but I doubt such useful functionality which is usable on various Posixen is more disturbing than Windows-only suprpressConsole https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954 I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.) Depends on the point of view. In my opinion both suppressConsole and posixCallack defines process details after process space is created. The only difference is that suppressConsole is an exceptional switch to define behavior on exactly one platform (Windows) while posixCallback allows defying behavior on multiple POSIX compatible systems. But I was mistaken. Config is an enum not struct, so yeah, not worth changing it only for sake of posix callback. So maybe module level variable? module std.process; // ... void delegate() posixPostFork = null; // ... Global state? Don't want to go there... Just proposition. I don't know what way you and Steven prefer. I would *really* love to see this implemented. It is really basic stuff for posixen. It needs a good API and community support. I don't think we should introduce new functionality, that looks like it was bolted on, because one person said they really needed it. I understand. I just didn't except that this will be so controversial. I thought that chopping off a lot of potential functionality on POSIX was just oversight, given that Windows has its own specific flag. If such functionality isn't needed and I am the only one whining about it, let's just forget about whole topic. I can live with my own more specialized implementation, I just thought it may be useful for others. Is it possible to abstract the things you would like to do in such a callback? You mention privilege lowering as a use case. My use cases are privilege lowering and setting up process limits (on per process basics). But my point was that post-fork callback opens the window of additional process tuning and other possibilities, like dumping memory map or whatever user may want to do. Can we make an API that does this, and which modifies the process' security context in Windows in an equivalent/similar way, for instance? I think we can, but then it closes various opportunities on POSIX. I'm not that much experienced with Windows to be honest, but I think it is possible to switch user, but not possible to set per process limits (at least not possible on non-Server versions of Windows). Again, I am not forcing anything on you or Steven. I am asking you for opinion on something I find useful because you are the experts and I really like your work on new std.process Best regards, Damian Ziemba
Re: std.process - POSIX specific callback
are you talking about http://linux.die.net/man/3/pthread_atfork
funktionality?
Am 06.06.2013 18:05, schrieb nazriel:
Would it be possible to add to std.process.Config POSIX specific
callback which would be called after fork()?
It is currently main blocker in switching dpaste-be from handmade
process handling module to std.process.
It could look something like this.
struct Config {
// current fields
void delegate() posixCallback;
}
// ...
int i = fork();
if (i > 0)
{
//...
if (config.posixCallback !is null)
config.posixCallback();
//...
}
Such construct would allow for various child process
manipulation, for instance
dropping root privileges or setting limits via setrmlimit.
Example:
config.posixCallback = {
setguid(ourGUID);
setgroups(ourGROUPS);
setuid(ourUID);
setrmlimit(NFORK, 123);
};
AFAIK we already have Windows specific flag related to spawning
console for GUI apps.
I can make pull request ASAP when I get reasonable name for field.
Lars? ;)
Re: std.process - POSIX specific callback
On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote: I am aware that std.process is generalized but I doubt such useful functionality which is usable on various Posixen is more disturbing than Windows-only suprpressConsole https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954 I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.) But I was mistaken. Config is an enum not struct, so yeah, not worth changing it only for sake of posix callback. So maybe module level variable? module std.process; // ... void delegate() posixPostFork = null; // ... Global state? Don't want to go there... I would *really* love to see this implemented. It is really basic stuff for posixen. It needs a good API and community support. I don't think we should introduce new functionality, that looks like it was bolted on, because one person said they really needed it. Is it possible to abstract the things you would like to do in such a callback? You mention privilege lowering as a use case. Can we make an API that does this, and which modifies the process' security context in Windows in an equivalent/similar way, for instance?
Re: std.process - POSIX specific callback
On Thursday, 6 June 2013 at 17:18:20 UTC, Steven Schveighoffer
wrote:
On Thu, 06 Jun 2013 12:05:14 -0400, nazriel
wrote:
Would it be possible to add to std.process.Config POSIX
specific callback which would be called after fork()?
It is currently main blocker in switching dpaste-be from
handmade process handling module to std.process.
It could look something like this.
struct Config {
// current fields
void delegate() posixCallback;
}
// ...
int i = fork();
if (i > 0)
{
//...
if (config.posixCallback !is null)
config.posixCallback();
//...
}
Such construct would allow for various child process
manipulation, for instance
dropping root privileges or setting limits via setrmlimit.
Example:
config.posixCallback = {
setguid(ourGUID);
setgroups(ourGROUPS);
setuid(ourUID);
setrmlimit(NFORK, 123);
};
AFAIK we already have Windows specific flag related to
spawning console for GUI apps.
I can make pull request ASAP when I get reasonable name for
field.
Lars? ;)
I agree with the ability, but not with the interface. If this
is to be done, it should be at a lower level, not inside
config. Keep in mind that std.process is generalized for both
Windows and Posix, with very minor differences.
-Steve
I am aware that std.process is generalized but I doubt such
useful functionality which is usable on various Posixen is more
disturbing than Windows-only suprpressConsole
https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954
But I was mistaken. Config is an enum not struct, so yeah, not
worth changing it only for sake of posix callback.
So maybe module level variable?
module std.process;
// ...
void delegate() posixPostFork = null;
// ...
I would *really* love to see this implemented. It is really basic
stuff for posixen.
Thanks a lot for responding Steven.
Re: std.process - POSIX specific callback
On Thu, 06 Jun 2013 12:05:14 -0400, nazriel wrote:
Would it be possible to add to std.process.Config POSIX specific
callback which would be called after fork()?
It is currently main blocker in switching dpaste-be from handmade
process handling module to std.process.
It could look something like this.
struct Config {
// current fields
void delegate() posixCallback;
}
// ...
int i = fork();
if (i > 0)
{
//...
if (config.posixCallback !is null)
config.posixCallback();
//...
}
Such construct would allow for various child process manipulation, for
instance
dropping root privileges or setting limits via setrmlimit.
Example:
config.posixCallback = {
setguid(ourGUID);
setgroups(ourGROUPS);
setuid(ourUID);
setrmlimit(NFORK, 123);
};
AFAIK we already have Windows specific flag related to spawning console
for GUI apps.
I can make pull request ASAP when I get reasonable name for field.
Lars? ;)
I agree with the ability, but not with the interface. If this is to be
done, it should be at a lower level, not inside config. Keep in mind that
std.process is generalized for both Windows and Posix, with very minor
differences.
-Steve
