Hi All, Happy new year to everyone!
I am still in the middle of moving the polling stuff over to a new machine. I'm still struggling with this one. It should be fairly trivial, but so far it is getting the best of me. I'm going to share some details concerning the problem that I am having, just in case someone else has seen this behavior, and because I have found that I can often solve problems by explaining them. So, the Syndic8 poller consist of a single master process and a variable number (currently 20) slave processes, all implemented in PHP. The master does a bunch of preparatory work and then writes the Feed Ids of the feeds to be polled to a message queue. The slaves are all reading from the queue. The slaves are in a tight loop which basically fetches the feed's XML, does some parsing and other checking, and then stores it into a file. Right now this process takes around 13 hours for about 350,000 feeds. On the new machine, things start out fine. Each slave processes thousands of feeds and the machine is respectably busy. Many hours into the process, something slowly goes awry, one slave at a time. Deep inside of PHP some memory is overwritten, and a normal "realloc" request turns into a request for some huge amount of memory. Slave after slave consumes a ton of memory, until all of the system's swap space is depleted. At this point the Linux "Out of Memory Killer" swings into action, indiscriminately axing processes left and right. At some point it ends up killing essential system processes such as the SSH demon, and the machine becomes inaccessible. When this happens, I get out of memory errors from deep inside of PHP's memory allocator: FATAL: erealloc(): Unable to allocate 1512939521 bytes FATAL: erealloc(): Unable to allocate 1809541121 bytes FATAL: erealloc(): Unable to allocate 1932369921 bytes It seems to me that some memory is getting overwritten, resulting in those giant sizes. I have printed them out as characters, and didn't get anything interesting; this is not a simple case of a string overwrite. I've rebuilt PHP in debug mode, and I have added more debug messages of my own. I have run up to 5 of the slaves inside of GDB, with a breakpoint at the point where it runs of out memory. I have not yet been able to hit this breakpoint. I have built and tested multiple versions of PHP (currently using 4.3.9, which runs just fine on the other machine), and CuRL. The behavior doesn't seem to change. I love to play detective, but this one almost has me stumped. My next thought is to configure in enough swap to allow all of the bogus allocations to succeed, so that I can get in deeper with the debugger. Another option is to start paring down the poller until I have the smallest possible code which will crash. Any thoughts or creative ideas for solving this would be greatly appreciated! Jeff; Jeff Barr wrote: > Hi Everyone, > > I am in the process of moving the Syndic8 polling system to a machine of > its own. When this is complete, the site should be faster, and the > polling itself should run in less time (currently it takes about 14 > hours to poll 375,000 or so feeds). In anticipation of this change, I > turned off polling a few days ago. > > Unfortunately, the new machine seems to have some sort of hardware > problem, which I am working with the hosting facility to resolve. > > In the interim I have turned the old poller back on. Once the new one is > running I hope that the main server has sufficient "headroom" to let us > build some cool new features in 2006. > > Jeff; > -- * RSS Feeds: http://www.syndic8.com * Blog: http://www.syndic8.com/~jeff/blog/ * Web Services: http://aws.typepad.com * Resume: http://www.syndic8.com/~jeff/resume.html If you're getting "too much mail" then send a message to: [EMAIL PROTECTED] (this will tell it to stop sending you mail) To change your membership use THIS WEB PAGE: http://finance.groups.yahoo.com/group/syndic8/join (here you can set your mail to digest, no mail or LEAVE the list) To find more info about Syndicated XML newsfeeds visit http://www.syndic8.com/ Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/syndic8/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
