Re: [Sks-devel] SKS RAM usage gone haywire

2009-03-22 Thread Kim Minh Kaplan
I was seeing this in recon.log: 2009-03-22 08:34:10 Error getting missing keys: Out of memory This is because the server makes use of the response even though the request failed. It then proceed onto erronously alloc huge amount of memory (which sometimes fail). On most systems this is not

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-16 Thread Ryan Hunt
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I don't know what is wrong with my system, I rebuilt the DB from my old dump.. brought it online and got updated from my peers... and then get stuck in a new loop trying to get new keys. I have disabled rcon at the moment to keep from hurting my

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-15 Thread Ari Trachtenberg
From memory, the theory predicts that recon running time and memory should grow roughly linearly with delta; only interaction should increase. In practice, everything depends on the actual implementation and how careful OCAML is in reclaiming unused memory. Maybe Yaron can chime in.

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-15 Thread Yaron Minsky
On Sat, Feb 14, 2009 at 6:14 PM, Phil Pennock sks-devel-p...@spodhuis.orgwrote: I begin to wonder if recon is sub-optimal with a large delta of keys to send and also to wonder if I should bump learn to read OCaml up my priority list -- I'm managing to navigate the sks source faster already,

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-15 Thread Phil Pennock
On 2009-02-15 at 10:05 -0500, Yaron Minsky wrote: Ari is right: there's nothing inherent about the algorithm that should require an ever-growing use of memory. OCaml itself is very careful about reclaiming unreferenced memory, but that of course does not preclude a memory leak in the code.

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-15 Thread Daniel Kahn Gillmor
On 02/15/2009 06:33 PM, Ryan wrote: Yea, I am pretty sure I am the one going haywire.. sorry guys, trying to figure out exactly what the hell is wrong... Thanks for the quick and excellent detective work, Phil and Ryan! Since i restarted sks on zimmermann.mayfirst.org, and Ryan took his system

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-15 Thread John Marshall
On Sun, 15 Feb 2009, 19:19 -0500, Daniel Kahn Gillmor wrote: I agree with Ryan here that even if his host *had* been compromised and doing what amounts to a gossip DoS attack, the other nodes in the network should not have tried to gobble up all the RAM on their respective systems. We should

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-14 Thread Daniel Kahn Gillmor
On 02/13/2009 10:21 PM, Phil Pennock wrote: Normally, sks consumes a negligible amount of the resources on my server; some GB of disk, some MB RAM (~45 right now, after restart), not enough network that I've bothered to isolate figures for it. I just found that my box was thrashing badly

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-14 Thread Phil Pennock
On 2009-02-14 at 16:50 -0500, Daniel Kahn Gillmor wrote: Yikes! Thanks for pointing that out, because you made me check up on a keyserver i'm responsible for. It looks like zimmermann.mayfirst.org is doing the same thing. the sks recon process in particular now has an RSS of 3.3g. :-(

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-14 Thread Daniel Kahn Gillmor
On 02/14/2009 05:24 PM, Phil Pennock wrote: So, the options behind it, if the correlation is more than coincidence, seem to be: 1 bad SKS update 2 bad query hitting pool.sks-keyservers.net 3 someone hitting the servers individually deliberately 4 someone doing full sync of a keyserver

Re: [Sks-devel] SKS RAM usage gone haywire

2009-02-14 Thread Phil Pennock
On 2009-02-14 at 17:45 -0500, Daniel Kahn Gillmor wrote: Maybe someone who knows the source and/or is proficient with the use of valgrind could assess whether sks recon is actually leaky? I had been running without *noticing* any increase for some time and am inclined to believe that it's a