Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Phil Pennock
On 2009-03-22 at 22:40 -0400, Yaron Minsky wrote:
> 2009/3/22 Phil Pennock 
> > The gotcha here is async DNS support in O'Caml.
> 
> I'm sure I'm missing something here, but why is asynchronous DNS a
> requirement?  Why not do the DNS queries using threads?

You're not missing anything, I was.  I was sticking to the
event-handling stuff already done in SKS and I know nothing about how
cleanly O'Caml lets you use threads and combine the two models.  If it's
easy and clean, then my oversight was ... glaring.

None of the current membership has any kind of locking against
concurrent updates -- how much is needed, with O'Caml, and how much
locking done automatically?

-Phil


___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Daniel Kahn Gillmor
On 03/22/2009 10:29 PM, Yaron Minsky wrote:
> I'm really confused.  People have piped in in both directions on this one,
> so does someone have the definitive story?  Is 1.0.10 the one that behaves
> correctly, or 1.0.9?

So far i haven't heard anyone claim that 1.0.10 works correctly.  1.1.0
works correctly, and david shaw just pointed out that 1.0.9 works
correctly.  I believe 1.0.10 is the only version with this particular bug.

> And yes, we should get a 1.0.11 release out soon.  I was waiting for the
> IPv6 patch to settle down and for everyone to agree that it worked for IPv4
> and IPv6 installations alike.


do you mean you're hoping to release 1.1.1 soon?  Or is there some sort
of branched development process going on?

--dkg



signature.asc
Description: OpenPGP digital signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Yaron Minsky
2009/3/22 Phil Pennock 

> The gotcha here is async DNS support in O'Caml.


I'm sure I'm missing something here, but why is asynchronous DNS a
requirement?  Why not do the DNS queries using threads?

y
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Yaron Minsky
I'm really confused.  People have piped in in both directions on this one,
so does someone have the definitive story?  Is 1.0.10 the one that behaves
correctly, or 1.0.9?

And yes, we should get a 1.0.11 release out soon.  I was waiting for the
IPv6 patch to settle down and for everyone to agree that it worked for IPv4
and IPv6 installations alike.

y

2009/3/22 Daniel Kahn Gillmor 

> On 03/22/2009 06:41 PM, David Shaw wrote:
> > The 'exact=on' problem is specific to 1.0.10.  It worked properly in
> 1.0.9.
> >
> > See: http://www.mail-archive.com/sks-devel@nongnu.org/msg00287.html
>
> Ah, thanks for the pointer, David.
>
> Given that this causes problems for users of gnupg, has any thought been
> given to requiring members of the keyserver pools to not run that
> version of SKS?  keys.gnupg.net itself contains several keyservers
> running 1.0.10, which misbehave in response to standard gpg searches by
> keyid.
>
>--dkg
>
>
> ___
> Sks-devel mailing list
> Sks-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/sks-devel
>
>
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Phil Pennock
On 2009-03-22 at 13:45 +, Kim Minh Kaplan wrote:
> I see what you mean here...  Except that periodic DNS lookups are *not*
> The Right Thing.  This is one area where I think SKS got it wrong: it
> should call out to the resolver each time it needs to connect to a
> server and let the caching happen in normal ways (DNS TTL).  Please have
> a look at my other message "Keep DNS mappings fresh"[1]

Oh, right, sorry -- I forgot about that, because it was incompatible
as-is with the IPv6 work and so I didn't apply it.

> With my patch the "additional load" is bigger but it will still be
> minuscule when compared to the rest of the traffic needed for the
> reconciliation protocol anyway.  This is not where we should look for
> optimization.

You're quite right -- relying on a functioning DNS cache is the correct
way to go (but see below) -- I was going for the quick and easy solution
(and feeling uncomfortable while doing so).

However, either the membership_reload_interval option needs to be
completely removed or the reconserver.ml needs to support it -- leaving
it as a dbserver-only option seems sub-optimal.

Call it paranoia resulting from maintaining mail-server code in previous
employment, where mtime collisions in a cluster were possible so relying
on mtime-changed was a bad plan.

So, "sks-mshp-timed2.patch" should probably go in -- it fixes the
mailsync reload (as both those patches do) and adds the event handler
for reload.  With the default reload interval of 5 hours, the load
addition is minimal (understatement) and the benefit is that you gain
assurance that the file change *will* be picked up.  Eventually.

> OTOH if the membership reload takes more than the gossip_interval and
> reconciliation_config_timeout setting (typically one minute) then the
> loading never finishes and the server never reconciles.  It happened to
> me when three of my partners' nameservers went out of service.  Making
> the lookup as needed solves this problem.

Yes, the reliance upon functional DNS is good.  But not the looking up
in the main flow of control.  This is where things get very sticky very
quickly.

As is, doesn't your patch lead to a recon connection from a non-peer
while one of your peers is without DNS being a mini-DoS attack?  So once
you have a peer with bad DNS, you become susceptible to recon service
DDoS?

When there's no DNS for a peer, and you try to find the DNS, then the
local DNS cache will respond quickly for a period of time which is the
negative cache TTL imposed by that server for SERVFAIL caching.  So you
go from the old scenario, where you're hung up every
membership_reload_interval/mtime-changed period, which is O(hours) to
hung up every negative cache-entry TTL expiry, which is O(minutes).

Provided you only get recon connections from peers, this only bites when
you get gossip from a peer with bad DNS.  Which isn't going to be too
often, but still more often than the old reload interval.  If you also
get recon connections from non-peers, suddenly your recon thread is hung
up at the whim of anyone willing to issue a connection every few
minutes.  Fortunately, the level of impact only scales up with the
number of peers with bad DNS, so you'll still *mostly* be serving.

Thus while your patch is clearly trying to do the right thing, I think
it's a step backwards in resilience.  (One more than offset by your
memory usage stability patch, but still ...)

The clearest way out of this is to require dbserver/reconserver to have
event handler callbacks for DNS, use asynchronous DNS callback
resolution; populate membership with None entries and at load/reload
fire off lookup for these.  During connection check, if an IP entry is
None and the last reload was more than N seconds ago (!Settings knob,
default to 3. ?) then (1) fire off another async DNS resolution and then
(2) return failure immediately, so that the peer gets penalised for
flaky DNS and your server isn't hanging in the main flow of control.

The gotcha here is async DNS support in O'Caml.

I found an announcement for an O'Caml async DNS library called netdns:
  
http://groups.google.com/group/fa.caml/browse_thread/thread/7bd2ae0a9415340d?pli=1
  http://oss.wink.com/netdns/
which is BSD-licensed and at version 0.1.  It's main documented
incompleteness s that it requires a full resolver -- which is what we
want here anyway.

In addition, from looking at it: it doesn't support  records it uses
incremental xids and I think it's using a constant source port, so you'd
really want to be using a localhost resolver; even then, since it's not
matching source port, you're vulnerable.  I don't think this library is
ready for use.

There's then "adns", which is GPL'd, with bindings in some languages but
not O'Caml (does include Haskell); "c-ares" which is BSD licensed, does
include IPv6 support, is widely used and I'm pretty sure it will have
dealt with the xid/port attacks.  There are various smaller libraries
too, which don't manage to keep t

Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Daniel Kahn Gillmor
On 03/22/2009 06:41 PM, David Shaw wrote:
> The 'exact=on' problem is specific to 1.0.10.  It worked properly in 1.0.9.
> 
> See: http://www.mail-archive.com/sks-devel@nongnu.org/msg00287.html

Ah, thanks for the pointer, David.

Given that this causes problems for users of gnupg, has any thought been
given to requiring members of the keyserver pools to not run that
version of SKS?  keys.gnupg.net itself contains several keyservers
running 1.0.10, which misbehave in response to standard gpg searches by
keyid.

--dkg



signature.asc
Description: OpenPGP digital signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread David Shaw

On Mar 22, 2009, at 11:08 AM, Daniel Kahn Gillmor wrote:

This makes me think that what we're seeing is a bug in older  
versions of

SKS that could cause serious incompatibilities.  The reason i found it
was a report from a user who was having difficulty searching for keys
from the keyservers by keyid.


The 'exact=on' problem is specific to 1.0.10.  It worked properly in  
1.0.9.


See: http://www.mail-archive.com/sks-devel@nongnu.org/msg00287.html

David



___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Phil Pennock
On 2009-03-22 at 11:08 -0400, Daniel Kahn Gillmor wrote:
> This makes me think that what we're seeing is a bug in older versions of
> SKS that could cause serious incompatibilities.  The reason i found it
> was a report from a user who was having difficulty searching for keys
> from the keyservers by keyid.

Changelog for 1.1.0 contains:
  - Some small changes to index view

Since this has gone from not working to working, it looks like change in
the right direction.  With some recent activity providing Yaron with the
asked-for other maintenance, we might even see a 1.1.1 release sometime
soon, with the memory consumption fix, dump fix, IPv6 support,
membership reload fix, etc, making it an upgrade that we can nudge many
operators to upgrade to.

Provided that people *without* IPv6 support confirm that with the two
IPv6 patches it still works once they "disable_ipv6:" in their config
files.

-Phil


pgp2NxTtx9Bsn.pgp
Description: PGP signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Daniel Kahn Gillmor
On 03/22/2009 09:02 AM, Kim Minh Kaplan wrote:
> Daniel Kahn Gillmor:
> 
>> gpg generates an HTTP request like this:
>>
>> http://$foo:11371/pks/lookup?op=index&options=mr&search=0xD21739E9&exact=on
> [...]
>> What is the right way to handle this?
> 
> The simplest solution would be to remove the "exact=on" parameter.

That may work, but:

 * it's gpg generating that query, not me by hand, so i can't easily
change it (and i certainly can't change it for everyone who i want to
support).

 * the exact same query (with exact=on) *works* against SKS 1.1.0, and
also against pgp.mit.edu, which reports itself as "pks_www/0.9.6"

This makes me think that what we're seeing is a bug in older versions of
SKS that could cause serious incompatibilities.  The reason i found it
was a report from a user who was having difficulty searching for keys
from the keyservers by keyid.

--dkg



signature.asc
Description: OpenPGP digital signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] SKS RAM usage gone haywire

2009-03-22 Thread Kim Minh Kaplan
I was seeing this in recon.log:

2009-03-22 08:34:10 Error getting missing keys: Out of memory

This is because the server makes use of the response even though the
request failed.  It then proceed onto erronously alloc huge amount of
memory (which sometimes fail).  On most systems this is not very serious
as the memory is never used so never really alloced.  This fixes it,
i.e. it does not try to use the result of failed requests.

Kim Minh.

diff -r 2cb760c4b34b -r 809482afe9f4 reconComm.ml
--- a/reconComm.ml	Sun Mar 22 09:33:32 2009 +
+++ b/reconComm.ml	Sun Mar 22 14:06:49 2009 +
@@ -84,7 +84,11 @@
  (String.length msg));
 		cout#write_string msg;
 		cout#flush;
-		ignore (input_line cin#inchan); (* read "HTTP" line *)
+		(* read "HTTP" line and make sure the status is 2xx *)
+		let status_re = Str.regexp "^HTTP/[0-9]+\\.[0-9]+ 2" in
+		let status = input_line cin#inchan in
+		if not (Str.string_match status_re status 0) then
+		  failwith status;
 		let _headers = Wserver.parse_headers Map.empty cin#inchan in
 		let keystrings = 
 		  CMarshal.unmarshal_list ~f:CMarshal.unmarshal_string cin
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Kim Minh Kaplan
Phil Pennock:

> 8< cut here >8--
> =item -membership_reload_interval
>
> Maximum interval (in hours) at which membership file is reloaded.
> 8< cut here >8--
>
> There are supposed to be *two* triggers for reload -- one is mtime
> change, one is -membership_reload_interval expiring.  This works in the
> dbserver, not in the reconserver.  Ironic.
>
> Thus I stand by the intent of my patches.
>
> With this fixed, people who change DNS won't have to post to lists
> asking people to touch their membership files to pick up the DNS change,
> and I will be able to get rid of my cron job which touches the
> membership file, as sks will do The Right Thing.

I see what you mean here...  Except that periodic DNS lookups are *not*
The Right Thing.  This is one area where I think SKS got it wrong: it
should call out to the resolver each time it needs to connect to a
server and let the caching happen in normal ways (DNS TTL).  Please have
a look at my other message "Keep DNS mappings fresh"[1]

> The additional load is an extra couple of DNS lookups per peer, every
> reload interval; it's a float, but I doubt many people set this to less
> than 1, so it's pretty light.

With my patch the "additional load" is bigger but it will still be
minuscule when compared to the rest of the traffic needed for the
reconciliation protocol anyway.  This is not where we should look for
optimization.

OTOH if the membership reload takes more than the gossip_interval and
reconciliation_config_timeout setting (typically one minute) then the
loading never finishes and the server never reconciles.  It happened to
me when three of my partners' nameservers went out of service.  Making
the lookup as needed solves this problem.

Regards,
Kim Minh.
[1] http://lists.gnu.org/archive/html/sks-devel/2009-03/msg00085.html


___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Phil Pennock
On 2009-03-22 at 12:47 +, Kim Minh Kaplan wrote:
> Phil Pennock:
> 
> > Previously, membership was only automatically reloaded in the db server,
> > not the recon server.
> 
> Why do you say this?  Reading Membership.get and Membership.test, the
> file is reloaded whenever it is modified.  thus this patch seems wrong.

sks.pod:
8< cut here >8--
=item -membership_reload_interval

Maximum interval (in hours) at which membership file is reloaded.
8< cut here >8--

There are supposed to be *two* triggers for reload -- one is mtime
change, one is -membership_reload_interval expiring.  This works in the
dbserver, not in the reconserver.  Ironic.

Thus I stand by the intent of my patches.

With this fixed, people who change DNS won't have to post to lists
asking people to touch their membership files to pick up the DNS change,
and I will be able to get rid of my cron job which touches the
membership file, as sks will do The Right Thing.

The additional load is an extra couple of DNS lookups per peer, every
reload interval; it's a float, but I doubt many people set this to less
than 1, so it's pretty light.

-Phil


pgph5R43fVXkf.pgp
Description: PGP signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] problems with SKS 1.0.10 when searching by key ID from GnuPG

2009-03-22 Thread Kim Minh Kaplan
Daniel Kahn Gillmor:

> gpg generates an HTTP request like this:
>
> http://$foo:11371/pks/lookup?op=index&options=mr&search=0xD21739E9&exact=on
[...]
> What is the right way to handle this?

The simplest solution would be to remove the "exact=on" parameter.

Kim Minh.


___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Kim Minh Kaplan
Phil Pennock:

> Previously, membership was only automatically reloaded in the db server,
> not the recon server.

Why do you say this?  Reading Membership.get and Membership.test, the
file is reloaded whenever it is modified.  thus this patch seems wrong.

Kim Minh.


___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] Dump recurses to dumpsize on stack?

2009-03-22 Thread Yaron Minsky
Man, that's an embarrassing mistake.  (In my defense, i did write that code
a long time ago.)  Kim's patch is precisely right.

y

2009/3/22 Kim Minh Kaplan 
>

> [ apologies if you see this twice I had put a wrong address in the first
> one ]
>
> Phil Pennock:
>
> > Is it known that when you do "sks dump  ", you end up with up to
> > 10 stack frames per N, plus the calls into the DB layer, etc?
> [...]
> > I was suspecting db problems, and it's not.  Looks as though something
> > was expecting tail recursion optimisation but not getting it.  I really
> > don't know enough about programming in functional languages to look into
> > this properly.  I don't even know if things like shuffling exception
> > handling around would let the tail recursion optimisation kick in and
> > where things like integrity of backup dumps are concerned, I play around
> > less.
>
> I highly suspect that the inside of a try block is not a tail position
> and that "shuffling exception handling around" is the right thing to do
> here.  Try this.
>
> Kim Minh.
>
> ___
> Sks-devel mailing list
> Sks-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/sks-devel
>
>
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


[Sks-devel] [PATCH] auto-refresh membership DNS

2009-03-22 Thread Phil Pennock
Each of the attached compiles, which means more than it does in some
other languages.  I'm fairly confident I haven't messed up.  But there's
two approaches.

Previously, membership was only automatically reloaded in the db server,
not the recon server.

Patch sks-mshp-timed.patch makes membership reload independent of
whether or not the server's eventloop does anything special, so this
sort of oversight can't happen.  It's more intrusive, because it changes
how membership is stored.

Patch sks-mshp-timed2.patch is much simpler and just adds an event
handler to reconserver.ml, similar to that in dbserver.ml.

There are pros and cons to each.

Separately, I believe that the mailsync file was getting reloaded rather
often, if the mtime differed from that of the membership file, because
the wrong file's mtime was checked.  Both patches fix that.

Regards,
-Phil
diff --git a/membership.ml b/membership.ml
--- a/membership.ml
+++ b/membership.ml
@@ -31,7 +31,7 @@ exception Lookup_failure of string
 exception Malformed_entry of string
 exception Empty_line
 
-let membership = ref ([| |],-1.)
+let membership = ref ([| |],-1.,-1.)
 
 let whitespace = Str.regexp "[ \t]+"
 
@@ -161,6 +161,7 @@ let get_mtime fname =
   Unix.Unix_error _ -> None
 
 let load_membership fname = 
+  let time_now = Unix.time () in
   let file = open_in fname in
   protect ~f:(fun () -> 
 		let mshp = load_membership_file file in
@@ -169,12 +170,12 @@ let load_membership fname =
 		  plerror 2 "%s" 
 			("Unable to get mtime for membership. " ^
 			 "Failed to reload.")
-		  | Some mtime -> membership := (mshp,mtime)
+		  | Some mtime -> membership := (mshp,mtime,time_now)
 	 )
 ~finally:(fun () -> close_in file)
 
 let membership_string () = 
-  let (mshp,_) = !membership in
+  let (mshp,_,_) = !membership in
   let to_string (addr_list,str) =
 String.concat " " (
   "(" :: (
@@ -185,9 +186,19 @@ let membership_string () =
   "Membership: " ^ String.concat ~sep:", " strings
 
 
+let reload_if_old () =
+  let (mshp,_,last_loaded) = !membership in
+  let age = Unix.time () -. last_loaded in
+  if age > !Settings.membership_reload_time then
+( let fname = Lazy.force Settings.membership_file in
+  load_membership fname;
+  plerror 5 "%s" (membership_string ())
+)
+
 let reload_if_changed () = 
+  reload_if_old ();
   let fname = Lazy.force Settings.membership_file in
-  let (mshp,old_mtime) = !membership in
+  let (mshp,old_mtime,_) = !membership in
   match get_mtime fname with
 | None -> 
 	plerror 2 "%s" ("Unable to get mtime for membership file. " ^
@@ -202,7 +213,7 @@ let get_names () =
   let mshp = 
 if Sys.file_exists (Lazy.force Settings.membership_file) then (
   reload_if_changed ();
-  let (m,mtime) = !membership in 
+  let (m,mtime,ltime) = !membership in 
   m
 )
 else [| |]
@@ -211,14 +222,14 @@ let get_names () =
 
 
 let reset_membership_time () =
-  let (m,mtime) = !membership in
-  membership := (m,0.)
+  let (m,mtime,ltime) = !membership in
+  membership := (m,0.,0.)
 
 let get () = 
   let mshp = 
 if Sys.file_exists (Lazy.force Settings.membership_file) then (
   reload_if_changed ();
-  let (m,mtime) = !membership in 
+  let (m,mtime,ltime) = !membership in 
   m
 )
 else [| |]
@@ -238,7 +249,7 @@ let ipmatch_ip_in_socklist ip1 sockaddr_
 
 let test addr = 
   reload_if_changed ();
-  let (m,mtime) = !membership in
+  let (m,mtime,ltime) = !membership in
   let ip = match inet_addr_of_sockaddr addr with
 | None -> assert false
 | Some x -> x
@@ -282,7 +293,7 @@ let load_mailsync_partners fname =
 
 let reload_mailsync_if_changed () = 
   let fname = Lazy.force Settings.mailsync_file in
-  let (mshp,old_mtime) = !membership in
+  let (_,old_mtime) = !mailsync_partners in
   match get_mtime fname with
   None -> plerror 2 "%s" 
 	("Failed to find mtime, can't decide whether to" ^
diff --git a/membership.ml b/membership.ml
--- a/membership.ml
+++ b/membership.ml
@@ -282,7 +282,7 @@ let load_mailsync_partners fname =
 
 let reload_mailsync_if_changed () = 
   let fname = Lazy.force Settings.mailsync_file in
-  let (mshp,old_mtime) = !membership in
+  let (mshp,old_mtime) = !mailsync_partners in
   match get_mtime fname with
   None -> plerror 2 "%s" 
 	("Failed to find mtime, can't decide whether to" ^
diff --git a/reconserver.ml b/reconserver.ml
--- a/reconserver.ml
+++ b/reconserver.ml
@@ -354,6 +354,8 @@ struct
 Eventloop.evloop
   ( [ Eventloop.Event (0.0, Eventloop.Callback catchup) ]
 	@ (Ehandlers.repeat_forever_simple catchup_interval catchup)
+@ Ehandlers.repeat_forever_simple !Settings.membership_reload_time
+  Membership.reset_membership_time
 	@ (if !Settings.gossip 
 	   then Ehandlers.repeat_forever 
 	 ~jitter:0.1 (* 10% randomness in delay interval *)


pgpk1ATH1RqOv.pgp
Description: PGP signature
___
Sks-devel mailing l

[Sks-devel] dump advice?

2009-03-22 Thread Phil Pennock
I can now dump keys without segfaults (thanks, Kim).  I have some
questions for people who generate dumps already.

I've been shutting down sks db & recon while dumping; in practice, do
people find that "sks dump" is safe to run while the db is running?

If I want to switch sks to using one of my own dumps as the base,
instead of the dumps I downloaded some time ago, what's involved in
cutting over?  Do I need to "sks fastbuild" again, or is there a way to
switch over the dump directories and then discard the entries from the
KDB which are now on disk?  If I've shut down sks, done a dump, can I
just remove KDB (leaving PTree) and go from there?  Or "sks cleandb"?

I'm hoping to save myself a few hours of a rebuild if I get this wrong
(or recovering from filesystem snapshot).

If I don't get swamped, I'll provide a doc patch for the .pod with what
I find out from people and experimentation.

Thanks,
-Phil


pgpdxdf8oX6Zb.pgp
Description: PGP signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] Dump recurses to dumpsize on stack?

2009-03-22 Thread Phil Pennock
On 2009-03-22 at 06:08 +, Kim Minh Kaplan wrote:
> I highly suspect that the inside of a try block is not a tail position
> and that "shuffling exception handling around" is the right thing to do
> here.  Try this.

That works.

"FRED/" is the test backup, with a count of 100_000 instead of 20_000
in:
-rw-r--r--  1 sks  sks   28472825 Mar 22 05:15 2009-03-21/foo-.pgp
-rw-r--r--  1 sks  sks  140514967 Mar 22 09:16 FRED/foo-.pgp

Thanks.  :)
-Phil


pgpw28bmUdFbi.pgp
Description: PGP signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel


Re: [Sks-devel] Dump recurses to dumpsize on stack?

2009-03-22 Thread Kim Minh Kaplan
Phil Pennock:

> Is it known that when you do "sks dump  ", you end up with up to
> 10 stack frames per N, plus the calls into the DB layer, etc?
[...]
> I was suspecting db problems, and it's not.  Looks as though something
> was expecting tail recursion optimisation but not getting it.  I really
> don't know enough about programming in functional languages to look into
> this properly.  I don't even know if things like shuffling exception
> handling around would let the tail recursion optimisation kick in and
> where things like integrity of backup dumps are concerned, I play around
> less.

I highly suspect that the inside of a try block is not a tail position
and that "shuffling exception handling around" is the right thing to do
here.  Try this.

Kim Minh.
diff -r 181a19755969 sksdump.ml
--- a/sksdump.ml	Sat Mar 21 18:31:36 2009 +
+++ b/sksdump.ml	Sun Mar 22 05:58:10 2009 +
@@ -46,19 +46,22 @@
   match SStream.next stream with
 	| None -> ()
 	| Some (hash,string) ->
+	let remain =
 	try
 	  let skey = Keydb.skey_of_string string in
 	  if should_dump skey then
 		let keystring = Keydb.keystring_of_skey skey in
 		output_string cout keystring;
-		write_to_file (size - 1) stream cout
+		size - 1
 	  else
-		write_to_file size stream cout
+		size
 	with
 		e -> 
 		  eplerror 1 e "Failed attempt to extract key %s" 
 		  (KeyHash.hexify hash);
-		  write_to_file size stream cout
+		  size
+	in
+	write_to_file remain stream cout
 
 
   let write_to_fname size stream fname = 
___
Sks-devel mailing list
Sks-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/sks-devel