I have a large sized project that runs lots of WWW::Mechanize processes. usually 60-100 processes at a time, hitting the net and visiting sites.
Weve been having a problem for some time now and done just about all the Os or hardware analysis we can, and we are unsure what to suspect next but obviously Mechanize (or even perl 5.8.0) are suspects. Basically, after a variable amount of time that seems to exponentially decrease as the number of threads increases, the processes will hang in a 'D' state of uninterruptible sleep. those of you who know the linux kernel will know this is a really rare and really bad situation. only a reboot can free them up. the kernel scheduler at a low level is awaiting the availability of a resource that obviously is not available. it rarely happens in certain nfs situations or cd-rom situations... if you do a web search for the problem that is. we can easily reproduce it once a day just by firing up a lot of copies of our code, which is heavily Mechanized/libwww. We have solaris boxes, we have redhat boxes, both succumb to it. we now are testing freeBSD and openBSD boxes to see if they have it as well. we are running a new 2.6 kernel with the new improved scheduler which we had hoped would solve the problem but hasnt. Can anyone think of situations that, under load, Mechanize could generate requests to the kernel that would actually send it into this rebooting situation? this is a condition that the kernel should prevent obviously, but isnt. day by day the evidence is mounting that the problem is either in Mechanize or possibly perl... but we are running 5.6.0 on solaris and 5.8.0 on redhat-2.6 so it seems unlikely its perl. any ideas of what to look for?? -- gedanken
