On Saturday 09 February 2008 14:14, Jusa Saari wrote: > On Fri, 08 Feb 2008 23:34:52 +0000, Matthew Toseland wrote: > > > On Linux, during CPU intensive node activities - resuming requests, > > decoding or encoding a large splitfile etc - the threads that do the core > > of Freenet's work (the packet sender and packet receiver threads, request > > senders etc), get starved of CPU by the CPU-intensive threads doing the > > FEC decoding (etc). The result is the node is dramatically slowed down and > > stops accepting requests because of this (we use the average round trip > > time for a message as effectively a measure of system load). It takes a > > while to recover afterwards because we use an averager to smooth it out. > > > > In theory this shouldn't happen, because we set thread priorities: > > MAX_PRIORITY for important stuff, MIN_PRIORITY for FEC decodes etc. > > > > Unfortunately, while thread priorities are used on Windows, they are *not* > > used on Linux. Linux only supports thread priorities for realtime threads. > > Practically if you run two java threads on a single core system one with > > MAX and one with MIN priority, both will get the same amount of cpu time > > on average. We have tested this. > > > > Further, the fairness features in the scheduler in 2.6.23 don't seem to > > help matters very much. > > Having two tasks running at the same priority get the same amount of > CPU time on average is the very definition of a fair scheduler ;).
Yeah but I had thought the critical threads didn't use that much CPU. > > > Nextgens suggested we use the java realtime API. The only reference I > > found to it was extremely unhelpful: > > > > "Java RTS is only available from Sun's OEM Sales team, which has > > knowledgeable sales staff worldwide. Please contact them at this address. > > You can also call Sales at +1-800-786-0404, Prompt 1, Prompt 3." ( > > http://java.sun.com/javase/technologies/realtime/ ) > > > > In other words, RTSJ is only available for big-budget closed source > > embedded stuff. > > > > A further complication is that if the node is starved of CPU the wrapper > > will restart it, and if it continues to be starved of CPU the wrapper > > shuts it down; this is presumably detected by a thread running within the > > JVM. > > > > The first part of the solution is true request resuming: At the moment, on > > startup, every request has to pull every key it used from the datastore, > > decode each layer, and generally do a lot of work which has already been > > done but we didn't save. > > > > Beyond that, the only realistic option appears to be an external daemon > > running at a lower nice level to do CPU-intensive jobs and FEC encoding > > and decoding in particular. > > > > Thoughts? > > You implied that you're using multiple threads for these CPU-intensive > tasks. Don't do that. Use Runtime.availableProcessors() threads. This way > there's just a single CPU-heavy thread per processor, so it can't starve > the transfer threads. We do for FEC decoding/encoding, maybe that is enough once we sort out request resuming. > > Or have the CPU-heavy threads sleep 10ms between each step in the decode. > Also, are you absolutely certain that this is caused by genuine lack of > CPU power, and not the decoding threads holding some lock during their > work which blocks the IO threads ? > > However, making the decoding an external process is propably the worst > possible solution, because: > > 1) It adds the overhead of IPC to the process. If the computer is already > running out of CPU power, this will make a bad situation worse. > > 2) Increases resource (memory) consumption. You need queues on both Fred > and the decode daemon, and more threads total to handle the interaction > between them, unless you're willing to let threads block (or use > non-blocking IO). > > 3) Makes debugging harder. Are you going to run one external process per > node in a simulator ? > > 4) Having things which are dependent on each other run at different > priorities is almost _asking_ for strange and obscure bugs to occur. Think > about it: if the decode daemon is starved of CPU, will a thread in Fred > proper trying to submit data to it block ? And if enough threads block, > what happens to Fred ? Will the wrapper restart Fred, despite it not being > the problem (and the restart thus not solving anything ? If it restarts > the decode daemon instead, what happens to the decodes in process ? > > 5) Is ultimately just a hack around a bad design. According to the > statistics page, Fred uses 400 threads to handle 20 connections with a > total transfer rate of 30 kbps. Seriously, WTF ?!? :) Fully asynchronous designs are pointless at that speed, and any modern software has to be multi-threaded. Certainly we could get rid of some more threads relatively easily, but there are core logic threads which are written synchronously because the code is clearest that way. > > > We could ask Tom Marble (the senior java performance guy, at least he was > > last year) to open source RTSJ :) ... but it wouldn't be ready for a long > > time, even if sun did take the plunge (and there's every reason not to). > > If the limitation - the inability to set the priority per-thread - is in > the Linux kernel, then RTSJ can't really help it, so this would be useless. Linux can set priority per thread. But only realtime priority. Having said that, it *might* be possible to set a nice value (outside of the posix api), because linux (contrary to the POSIX spec) doesn't maintain a single nice value for each thread. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20080209/faa9f3a3/attachment.pgp>
