On Sat, Sep 16, 2006 at 11:17:59PM +0300, Jusa Saari wrote: > On Sat, 09 Sep 2006 16:05:56 +0100, toad wrote: > > > Saving a LOT of lines of code. Threads are entirely legitimate tools. > > Hundreds of threads is reasonable on modern chips, especially as they are > > likely double the number of cores every 2 years for some time to come. > > However, ***IF*** it proves to be a serious problem there is a lot we can > > do - later on - to reduce freenet's usage of threads. > > How much memory do the threads use ? AFAIK JVM defaults to operating > system stack size default, and that size is 2Mb in Linux.
It allocates it all up-front? Really? Why on earth would it do that? > This means that > a 100 threads use 200 megabytes of memory in stack space alone. Then > there's the matter of wildly varying routing time if these threads are > competing from the CPU simultaneously. Finally, in Linux each thread is > scheduled independently of any other thread of the same application, so > even if Freenet is running at nice 19, it will end up using the vast > majority of CPU time even if there's nice 0 programs runnable (a hundred > threads getting 1/20th of CPU time (compared to nice 0 application) each > add to 5 times as much CPU time as a nice 0 application). That's not true. Linux 2.6 has a very well behaved POSIX compliant threads implementation. > > Besides, having lots of threads becoming runnable or blocked at an > unsteady rythm will cause other programs to "hiccup" randomly. This, of > course, is really a Linux scheduler problem, Which went away in 2.6, and most people use 2.6 nowadays. > but it will discourage users > from leaving Freenet running nonetheless. And it will happen in *any* > machine, no matter how powerfull the CPU or how many cores there might be, > since a hundred threads is perfectly capable of bogging them all down. Only if they are ACTUALLY RUNNING. They won't get scheduled unless they have work to do. > > No, this is a bug. A design bug perhaps, but still a bug. I disagree. 1000 threads is a bug. 100 threads is acceptable. > The question is, > can it be solved at this time ? After all, getting rid of threads usually > means asynchronous IO, and that is both difficult, labor-intensive, and > error-prone. Still, just because fixing a bug is difficult or impossible > doesn't mean it isn't a bug. I still disagree. :) > > > We cannot avoid having *some* threads. That means we cannot avoid the bugs > > that come with threads. Which means there is nothing to be gained from a > > development time perspective from using fewer threads (except in specific > > cases e.g. there are many checkpointed threads that could be simplified to > > run on the ticker and use one less thread at the same time; these only > > make up 10 or so of the threads, so they're not a priority from a > > performance perspective). And there is much to be lost. State machines are > > usually far more complex, and far harder to debug. > > Since no one seems to be able to figure out where the 1000 threads are > coming from, I'd say that the current system is impossible to debug ;). Very funny. Nobody has seriously tried. We haven't even seen a stack dump. > > > On Thu, Sep 07, 2006 at 09:17:49PM +0000, Thomas Bruderer wrote: > >> > >> > > Someone who develops and tells me that 1000 threads for one single > >> > > application is normal... THATS what I am flameing about. > >> > > >> > That always depends on the application. If I have to write an > >> > application that solves a problem that warrants 1000 threads, why the > >> > hell not? And Freenet has a hell of a lot stuff to do. Using threads > >> > is a logical and fast way to solve a certain class of problems and > >> > there's nothing wrong with doing it that way. > >> > >> Logical? Only for programmers not knowing the gigantic overhead of > >> threads. Fast? Its probrably for most problems not even faster to > >> develop, especially since multithreading always leads to the nastiest > >> bug ever. And for the application its obvious that it gets slower with > >> each thread. > >> > >> There is much wrong with using so many threads. > >> > >> debugging impossible, still nobody can tell me where all those threads > >> come from. Thats a very good security aproach to make it undebuggable. > >> > >> How do you think are IRC Servers written? NO, they dont use a sheer > >> amount of threads. Even though in that case the benefit would be > >> obvious. You know why they dont use many threads? because they couldnt > >> serve thousands or tenths of thousands users simultanously. It would > >> break any reasonable machine. > >> > >> In particular in Freenet I dont know of any bigger Problem which needs > >> using multiple threads besides the different services, (fred, fcp), and > >> one thread per > >> connection. > >> > >> Maybe its too easy to write new Thread in java, maybe posix threads had > >> its value in its complexity. People using them knew how difficult they > >> are. > >> > >> > >> > as you have mentioned quite correctly - threads are indeed expensive. > >> > Unfortunately in most cases it's either "use a thread" or "write > >> > several hundred lines of code and not use a thread." > >> > >> Yes, I really got upset when I read the answer of toad. Maybe I was a > >> bit harsh. But I think this careless programming leads to more problems > >> than saving some hundred lines of code. > >> > >> > >> > >> > >> > >> _______________________________________________ Devl mailing list > >> Devl at freenetproject.org > >> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > >> _______________________________________________ Devl mailing list > > Devl at freenetproject.org > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > > > _______________________________________________ > Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20060918/ad8f02e1/attachment.pgp>
