On Fri, 08 Feb 2008 23:34:52 +0000, Matthew Toseland wrote: > On Linux, during CPU intensive node activities - resuming requests, > decoding or encoding a large splitfile etc - the threads that do the core > of Freenet's work (the packet sender and packet receiver threads, request > senders etc), get starved of CPU by the CPU-intensive threads doing the > FEC decoding (etc). The result is the node is dramatically slowed down and > stops accepting requests because of this (we use the average round trip > time for a message as effectively a measure of system load). It takes a > while to recover afterwards because we use an averager to smooth it out. > > In theory this shouldn't happen, because we set thread priorities: > MAX_PRIORITY for important stuff, MIN_PRIORITY for FEC decodes etc. > > Unfortunately, while thread priorities are used on Windows, they are *not* > used on Linux. Linux only supports thread priorities for realtime threads. > Practically if you run two java threads on a single core system one with > MAX and one with MIN priority, both will get the same amount of cpu time > on average. We have tested this. > > Further, the fairness features in the scheduler in 2.6.23 don't seem to > help matters very much.
Having two tasks running at the same priority get the same amount of CPU time on average is the very definition of a fair scheduler ;). > Nextgens suggested we use the java realtime API. The only reference I > found to it was extremely unhelpful: > > "Java RTS is only available from Sun's OEM Sales team, which has > knowledgeable sales staff worldwide. Please contact them at this address. > You can also call Sales at +1-800-786-0404, Prompt 1, Prompt 3." ( > http://java.sun.com/javase/technologies/realtime/ ) > > In other words, RTSJ is only available for big-budget closed source > embedded stuff. > > A further complication is that if the node is starved of CPU the wrapper > will restart it, and if it continues to be starved of CPU the wrapper > shuts it down; this is presumably detected by a thread running within the > JVM. > > The first part of the solution is true request resuming: At the moment, on > startup, every request has to pull every key it used from the datastore, > decode each layer, and generally do a lot of work which has already been > done but we didn't save. > > Beyond that, the only realistic option appears to be an external daemon > running at a lower nice level to do CPU-intensive jobs and FEC encoding > and decoding in particular. > > Thoughts? You implied that you're using multiple threads for these CPU-intensive tasks. Don't do that. Use Runtime.availableProcessors() threads. This way there's just a single CPU-heavy thread per processor, so it can't starve the transfer threads. Or have the CPU-heavy threads sleep 10ms between each step in the decode. Also, are you absolutely certain that this is caused by genuine lack of CPU power, and not the decoding threads holding some lock during their work which blocks the IO threads ? However, making the decoding an external process is propably the worst possible solution, because: 1) It adds the overhead of IPC to the process. If the computer is already running out of CPU power, this will make a bad situation worse. 2) Increases resource (memory) consumption. You need queues on both Fred and the decode daemon, and more threads total to handle the interaction between them, unless you're willing to let threads block (or use non-blocking IO). 3) Makes debugging harder. Are you going to run one external process per node in a simulator ? 4) Having things which are dependent on each other run at different priorities is almost _asking_ for strange and obscure bugs to occur. Think about it: if the decode daemon is starved of CPU, will a thread in Fred proper trying to submit data to it block ? And if enough threads block, what happens to Fred ? Will the wrapper restart Fred, despite it not being the problem (and the restart thus not solving anything ? If it restarts the decode daemon instead, what happens to the decodes in process ? 5) Is ultimately just a hack around a bad design. According to the statistics page, Fred uses 400 threads to handle 20 connections with a total transfer rate of 30 kbps. Seriously, WTF ?!? > We could ask Tom Marble (the senior java performance guy, at least he was > last year) to open source RTSJ :) ... but it wouldn't be ready for a long > time, even if sun did take the plunge (and there's every reason not to). If the limitation - the inability to set the priority per-thread - is in the Linux kernel, then RTSJ can't really help it, so this would be useless. > > > Sources: > http://kerneltrap.org/node/6080 > http://www.md.pp.ru/~eu/jdk6options.html#ThreadPriorityPolicy_______________________________________________ > Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
