Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
On Sunday 01 September 2002 18.50, Paul Davis wrote:
> [I wrote:]
> >The only thing the sound servers need to be able to handle
> >RT audio are:
> >* one of the RT scheduling methods.
> >* locked memory. (Not protected at the moment - hard limits?)
>
> >This can be archived with a wrapper that drops ALL other priorities
> >before running the real application.
>
> this is, alas, not true. to use RT scheduling, you need either root
> uid or CAP_RESOURCE. if you have CAP_RESOURCE or root uid, you can do
> many, many things (such as write to /dev/kmem) that can lead to a DOS.
So this code from artswrapper.c does not work then?
(hmm, shouldn't it should do setegid too...)
/* drop root privileges if running setuid root
(due to realtime priority stuff) */
if (geteuid() != getuid())
{
#if defined (HAVE_SETEUID) && !defined (HAVE_SETEUID_FAKE)
seteuid(getuid());
#else
setreuid(-1, getuid());
#endif
}
But we can play safer than that!
Since the priorities can be set by another process, we can let the monitor do
it! (In the same way as it can reduce the priorities). And if we do it this
way we can build a higher level interface - requesting C cycles or MACs every
T microseconds. Or is ther a better interface?
Problems:
* it is hard to lock the memory... since mlockall does not work
over forks...
* How to prevent ANY program to request this? Might not be a problem since we
will not give away more cycles than available...
This might be a very good thing - giving the ability to request RT prio for
any program.
>
> the central problem is that there is no way to tell the kernel "this
> thread should get to run whenever it wants, but do nothing else that a
> normal process cannot do", and likewise, "this vm address space can
> lock down up to 128MB of memory". we don't have sufficiently
> granularity to do this, so we have to grant overly broad access. that
> in turns means that anything granted such access has many, many more
> ways of attacking the system than burning CPU cycles or locking down
> too much memory (and, BTW, i'm not sure if this latter issue is a
> problem or not with the emergence of process killing logic in the
> kernel these days).
The OOM killer might be fooled into killing an other random process...
(I have to check the code to be sure what will happen in this unnormal
situation)
My feeling is that this should be handled by the monitor aswell.
Maybe by sharing locked memory with the processes or by introducing
a new system call. Anyway - on a computer with enough memory this
should not be a problem. (but it might since several kernel versions might
swap out pages rather than flushing a recently read or written file...)
>
> to make this situation better means recognizing the utility of
> granting very specific resources to be used or more or less
> exclusively by a single task, even when those resources more or less
> mean "all of the machine". this is not something that POSIX or Linux
> provides at this time.
>
> --p
>
>
I will give this monitor addition a try tonight.
/RogerL
--
Roger Larsson
Skellefteå
Sweden
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
> >The only thing the sound servers need to be able to handle > >RT audio are: > >* one of the RT scheduling methods. > >* locked memory. (Not protected at the moment - hard limits?) > > >This can be archived with a wrapper that drops ALL other priorities > >before running the real application. > > this is, alas, not true. to use RT scheduling, you need either root > uid or CAP_RESOURCE. if you have CAP_RESOURCE or root uid, you can do > many, many things (such as write to /dev/kmem) that can lead to a DOS. I think you need: - CAP_SYS_NICE for changing into RT_FIFO, but that also: /* Allow raising priority and setting priority on other (different UID) processes */ /* Allow use of FIFO and round-robin (realtime) scheduling on own processes and setting the scheduling algorithm used by another process. */ so yes, it is dangerous. - CAP_IPC_LOCK for locking down memory (mlock and mlockall) CAP_RESOURCE is only needed if you want to program the rtc clock with a rate higher than 64Hz. And yes, if you grant it then the process can override any limits in resource usage assigned to it, not just the rtc. It is very broad in what it can do. Jackstart (the small program that can start jackd with rt capabilities) also gets CAP_SETPCAP which enables it to give capabilities (a restricted set) to other processes. > the central problem is that there is no way to tell the kernel "this > thread should get to run whenever it wants, but do nothing else that a > normal process cannot do", and likewise, "this vm address space can > lock down up to 128MB of memory". we don't have sufficiently > granularity to do this Yes, capabilities are useful but way too broad. -- Fernando
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
>I noticed that I was not on the list... >But I have now read the archives. > >Kernel changes are not workable - at least not short testm! >It will take ages until that change is available on most computers. >(But if it is the only solution then it is...) > >Local DOS/security exploits has traditionally been seen as equally >dangerous in Unix land - computers often run on Universities... >This is the major reason for Linux to be secure even remote. >(if you can not trust the security for a user logged how can you >trust the security of the whole system if the web server is run as >a normal user?) >And remember this IS perceived as serious enough for the >developers of KDE to disable the RT possibility for arts! precisely. it *is* a real threat, and if someone wanted to try to attack using this kind of method, its going to be very, very hard to stop them, monitor or otherwise. >The only thing the sound servers need to be able to handle >RT audio are: >* one of the RT scheduling methods. >* locked memory. (Not protected at the moment - hard limits?) >This can be archived with a wrapper that drops ALL other priorities >before running the real application. this is, alas, not true. to use RT scheduling, you need either root uid or CAP_RESOURCE. if you have CAP_RESOURCE or root uid, you can do many, many things (such as write to /dev/kmem) that can lead to a DOS. the central problem is that there is no way to tell the kernel "this thread should get to run whenever it wants, but do nothing else that a normal process cannot do", and likewise, "this vm address space can lock down up to 128MB of memory". we don't have sufficiently granularity to do this, so we have to grant overly broad access. that in turns means that anything granted such access has many, many more ways of attacking the system than burning CPU cycles or locking down too much memory (and, BTW, i'm not sure if this latter issue is a problem or not with the emergence of process killing logic in the kernel these days). to make this situation better means recognizing the utility of granting very specific resources to be used or more or less exclusively by a single task, even when those resources more or less mean "all of the machine". this is not something that POSIX or Linux provides at this time. --p
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
I noticed that I was not on the list... But I have now read the archives. Kernel changes are not workable - at least not short testm! It will take ages until that change is available on most computers. (But if it is the only solution then it is...) Local DOS/security exploits has traditionally been seen as equally dangerous in Unix land - computers often run on Universities... This is the major reason for Linux to be secure even remote. (if you can not trust the security for a user logged how can you trust the security of the whole system if the web server is run as a normal user?) And remember this IS perceived as serious enough for the developers of KDE to disable the RT possibility for arts! The only thing the sound servers need to be able to handle RT audio are: * one of the RT scheduling methods. * locked memory. (Not protected at the moment - hard limits?) This can be archived with a wrapper that drops ALL other priorities before running the real application. - If that drop is done in a safe manner, we have come a long way. At that point the server with its plugins, can do two things a normal process can not: * can dry out the CPU * can lock up memory - force the system into an out of memory situation * can fool the monitor into doing something stupid I have tried to handle this issues in my monitor: * runs on higher priority than the RT audio application * only keeps info about RT processes - ignores others. * uses only static memory structures - no need to allocate unavailable memory... - harder to fool into allocating huge amounts of memory itself... * reduces RT priority of all RT processes at once - harder to hide the attacking process Things missing: * should use a random checking interval. * should also monitor the locked memory (RSS) of RT processes. * it should not reduce the priority of system RT processes since that might be the reason for the DOS attack. [contradicion!] * it should have a log for the system administrator to look in. /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
On Thu, Aug 29, 2002 at 09:12:07PM +0200, Stefan Westerfeld wrote: > I am very interested in that. In all the discussions we had > about the RT issue in aRts, things usually came to the point: > basically, it is a kernel bug, that as soon as you use RT prio, > your system becomes unstable. Especially if you combine that > with dynamically loaded modules, which even might be third > party, both approaches to work around the problem break. This is no kernel BUG, this is POSIX. People actually rely on POSIX in this regard and the kernel has to follow. If you work around Unix protection mechanisms by loading unknown code into the virtual memory image of your privileged application YOU are opening a security hole and cannot expect the kernel to protect you or your user. For the kernel you and the malicous code you just loaded as you module is EXACTLY THE SAME unless you give it a seperate process. So please stop pointing out kernel bugs, where the kernel is behaving as required by POSIX and people who use the POSIX features. Regards Ingo Oeser -- Science is what we can tell a computer. Art is everything else. --- D.E.Knuth
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
On Thu, Aug 29, 2002 at 05:45:09PM -0400, Paul Davis wrote:
> the bug-in-the-code case is handled using the monitor thread approach,
> since we know that the monitored threads run at lower priority.
assuming the bug isn't in the monitor thread itself (or which affects
it).
you can argue that you can make the monitor small enough that it's
unlikely, and fork it to protect its address space, and that the user
shouldn't do 'killall blah' and accidently kill the monitor before the
rest, but the all around more robust solution is a scheduler change.
> this is not a "small patch", believe me. it might end up being not
> much code, but i am certain that to be done correctly, it will have to
> be very subtle and probably very elegant.
you are probably correct and even if i fix it at all; i doubt it will be
elegant.
> trying to stop an intentional attack that uses RT priority to launch a
> DOS seems like a waste of time: the relevant process already has
> either CAP_RESOURCE and/or root priviledge, and can do more or less
> *anything* it wants already.
what i would like to do is allow SCHED_USER_FIFO to be a non-root
thing, since you won't have the potential to take down the machine
(although you can certainly arbitrarily affect performance). the win i
think would be not giving the program any capabilities in the first
place.
rob
Robert Melby
Georgia Institute of Technology, Atlanta Georgia, 30332
uucp: ...!{decvax,hplabs,ncar,purdue,rutgers}!gatech!prism!gt4255a
Internet: [EMAIL PROTECTED]
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
>> a side note: JACK, when run in RT mode, launches its own maximal >> priority thread to perform exactly this function. all other RT threads >> run at lower priorities. i believe that it is not possible to use JACK >> to perform DOS attacks like this unless the client modifies its >> scheduling priority itself. > >which is still an issue. relying on the courtesy of user programs is >probably not optimal (as i'm sure you're all well aware). > >this issue bugs me especially because people go on and on about the >stability of *nix, and yet here we have an easy way in which a user >(albeit root) program can bring the system to its knees. all it takes >is one small bug in a reasonably complex program--not exactly >uncommon. i just don't agree with this. there are two issues: 1) a bug in the code that causes a DOS to occur 2) an intentional DOS attack the bug-in-the-code case is handled using the monitor thread approach, since we know that the monitored threads run at lower priority. the intentional DOS attack can't be handled in any reasonable way to start with, since the DOS attack can shutdown the monitoring system without your suggested fix: >i've been working on a kernel patch that will allow a program to use >the entire cpu up to some percentage of cycles which are reserved. >(this is a common approach in RT literature). preliminary testing >seems to indicate its workable, but a number of issues need to be >cleared up (>1 cpu, what happens if more than one process wants fifo >scheduling, etc). i will probably add a new scheduling policy, >something like SCHED_USER_FIFO. this is not a "small patch", believe me. it might end up being not much code, but i am certain that to be done correctly, it will have to be very subtle and probably very elegant. trying to stop an intentional attack that uses RT priority to launch a DOS seems like a waste of time: the relevant process already has either CAP_RESOURCE and/or root priviledge, and can do more or less *anything* it wants already. --p
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
/wrote Stefan Westerfeld <[EMAIL PROTECTED]> [Thu, 29 Aug 2002 21:16:59 +0200] | Hi! | |On Wed, Aug 28, 2002 at 10:15:54AM -0400, Paul Davis wrote: |> a side note: JACK, when run in RT mode, launches its own maximal |> priority thread to perform exactly this function. all other RT threads |> run at lower priorities. i believe that it is not possible to use JACK |> to perform DOS attacks like this unless the client modifies its |> scheduling priority itself. | |As far as I understood this, you have a client thread with raised priority |that gets monitored. However, couldn't an attacker fork() in this thread, |to transport priviledges to another (unrelated) process, and then kill -9 |all other processes with priviledges, and then do his DOS attack? Anyway, what is the point of all this? I have nothing against security, but: 1. that dos vulnerability is a local one 2. local dos vulnerabilities are of importance for systems where "untrusted" users are roaming 2 doesn't seem like the typical setup where one would run a server dedicated to realtime audio like JACK, especially not in RT mode, as you'd have no guarantee anyway you'd have enough cpu for the many softsynths/audio software you'd want to play with.. Of course nothing forbids in an imaginary world to use jack as an esd/arts replacement, but it doesn't make sense then to run it in RT mode.
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
Hi! On Wed, Aug 28, 2002 at 10:15:54AM -0400, Paul Davis wrote: > a side note: JACK, when run in RT mode, launches its own maximal > priority thread to perform exactly this function. all other RT threads > run at lower priorities. i believe that it is not possible to use JACK > to perform DOS attacks like this unless the client modifies its > scheduling priority itself. As far as I understood this, you have a client thread with raised priority that gets monitored. However, couldn't an attacker fork() in this thread, to transport priviledges to another (unrelated) process, and then kill -9 all other processes with priviledges, and then do his DOS attack? Cu... Stefan -- -* Stefan Westerfeld, [EMAIL PROTECTED] (PGP!), Hamburg/Germany KDE Developer, project infos at http://space.twc.de/~stefan/kde *-
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
Hi! On Wed, Aug 28, 2002 at 12:47:41PM -0400, rm wrote: > which is still an issue. relying on the courtesy of user programs is > probably not optimal (as i'm sure you're all well aware). > > [...] > > i've been working on a kernel patch that will allow a program to use > the entire cpu up to some percentage of cycles which are reserved. > (this is a common approach in RT literature). preliminary testing > seems to indicate its workable, but a number of issues need to be > cleared up (>1 cpu, what happens if more than one process wants fifo > scheduling, etc). i will probably add a new scheduling policy, > something like SCHED_USER_FIFO. I am very interested in that. In all the discussions we had about the RT issue in aRts, things usually came to the point: basically, it is a kernel bug, that as soon as you use RT prio, your system becomes unstable. Especially if you combine that with dynamically loaded modules, which even might be third party, both approaches to work around the problem break. 1. if you try to write a monitoring system, then it is usually easy for the dynamically loaded code to somehow kill or fool the monitoring system, or to put it the other way round, it is hard to write a bullet-proof monitoring system 2. if you try to audit all code that gets loaded and executed with RT prio, and only load "trusted code", you're facing an endless task What I thought might be interesting as well, once you have such a scheduling policy, is a new ulimit that administrators can use to restrict RT prio on their system to i.e. max 95% cpu usage - that way, they don't need to adress all programs individually. That ulimit could even be there by default, so that RT stability issues would disappear regardless of how programs used RT prio in detail. Cu... Stefan -- -* Stefan Westerfeld, [EMAIL PROTECTED] (PGP!), Hamburg/Germany KDE Developer, project infos at http://space.twc.de/~stefan/kde *-
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
On Wed, Aug 28, 2002 at 10:15:54AM -0400, Paul Davis wrote:
> a side note: JACK, when run in RT mode, launches its own maximal
> priority thread to perform exactly this function. all other RT threads
> run at lower priorities. i believe that it is not possible to use JACK
> to perform DOS attacks like this unless the client modifies its
> scheduling priority itself.
which is still an issue. relying on the courtesy of user programs is
probably not optimal (as i'm sure you're all well aware).
this issue bugs me especially because people go on and on about the
stability of *nix, and yet here we have an easy way in which a user
(albeit root) program can bring the system to its knees. all it takes
is one small bug in a reasonably complex program--not exactly
uncommon.
i've been working on a kernel patch that will allow a program to use
the entire cpu up to some percentage of cycles which are reserved.
(this is a common approach in RT literature). preliminary testing
seems to indicate its workable, but a number of issues need to be
cleared up (>1 cpu, what happens if more than one process wants fifo
scheduling, etc). i will probably add a new scheduling policy,
something like SCHED_USER_FIFO.
anyway hopefully soonish i will get a chance to swap it in and make a
patch if anyone is interested. (if lots of folks are interested i'll
get to it sooner rather than later if possible).
if anyone has thoughts feel free to share.
rob
Robert Melby
Georgia Institute of Technology, Atlanta Georgia, 30332
uucp: ...!{decvax,hplabs,ncar,purdue,rutgers}!gatech!prism!gt4255a
Internet: [EMAIL PROTECTED]
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
>One problem with audio systems that >a) runs as RT processes >b) allows user loaded plugins > >Is that it is very simple to make a DOS (Denial of service) attack. >This has resulted in that the feature to run arts in RT mode has been >disabled... see artswrapper.c, search for NO_MORE_LOCAL_DOS_HOLE > >I made this code as a proof of concept for a monitor that catches and >reduces priority for processes that misuse this feature. >(this version reduces the priority for all RT processes at once...) a side note: JACK, when run in RT mode, launches its own maximal priority thread to perform exactly this function. all other RT threads run at lower priorities. i believe that it is not possible to use JACK to perform DOS attacks like this unless the client modifies its scheduling priority itself. --p
