On Friday 19 February 2010 10:04:06 alex wrote: > Ian Clarke wrote: > > > We do have a project license key for the excellent YourKit Profiler, > > which can help track down and diagnose both memory and CPU > > bottlenecks. It is also free to try for 30 days I believe. If you > > are seeing CPU usage or memory issues, might I suggest using YourKit > > and dumping a memory or CPU snapshot with it? > > I've tried in the past to hunt down memory leaks with jconsole/jmap but it's > not easy at all, and the code related to insertions is really hard for > somebody not familiar with it (or, well, at least it was for me). > > In my (dated) experience, memory problems were in the end caused by leaks > related to fetchs/insertions, and not to idle/sporadic browsing operation of > the node. But for some time I published an index site, and running the 24/7 > spider and the (weekly) insertion caused the node to crash regularly on OoM. > jconsole was quite explicit on the memory ramping up, although the spider > used a fixed number of fetchers. > > Too bad I never managed to narrow down to anything that could be reported > and be helpful in fixing something. I might try with the YourKit profiler if > I find some spare time.
Frankly, you are expected to have a supercomputer if you run a spider index... :| Anyway the next task re the spider is making it write the data out progressively linearly every hour rather than all at once with lots of seeking over a period of a week or more. This should greatly simplify the architecture and reduce the need for database usage. When it will happen I dunno, it should be pretty high priority though ... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100219/f2f84ff9/attachment.pgp>
