Re: 6 million
On 14-04-2020 21:35, brent s. wrote: > On 4/14/20 15:17, Stefan Claas wrote: >> brent s. wrote: >> >>> On 4/14/20 11:00, Stefan Claas wrote: >>>> >>>> Why still focusing on a dead project like SKS and not convining the other >>>> guys from Mailvelope or Hagrid to add peering capabilities? >>>> >>> >>> You do realize one can do both, right? >> >> Yes, and I have not seen here from the majority in the past, saying hey lets >> try out (and switch) or asked the devs. > > We can't switch because the "replacements" lack functionality SKS has. This kind of response killed many discussions we had in the past on this list about possible solutions for the problems of SKS. We saw the problems coming, sat back, watched while it happened and many have quit. That just makes me wonder who is the troll? Another good argument to end any remaining discussion was to state that the proposed solution would not prevent the full one hundred percent of problems. So, we never got to make the first step in hardening SKS to prevent abuse. These two methods proved to be very effective. If there was one multi-personality party (government?) bringing up these two arguments (sometimes repeatedly), with the objective to stall SKS development, then they must be laughing out lout, about how easy their job has been. > Until there is a complete replacement for SKS, SKS will continue to be > operated. Without stating the 'must have' requirements, but simply stating 'complete', the functionality causing the problems can never be removed or changed. Even trying to state the objective of sks-keyservers.net never succeeded, as every operator had their own objective to operate an SKS server. Therefore, trying to obtain the 'must have' requirements is a mission impossible by itself. Just my 2 cents. Operate whatever software you like. Kind regards, Arnold
Re: [Sks-devel] The pool is shrinking
I thought SKS and PGP-keys is about one's ability to hide private data (by encryption). GDPR is also about one's ability to hide private data (by having private data, that can be used in correlations, removed from large databases). Yet, SKS administrators who apparently live outside the EU argue strongly that there is no need for them to support GDPR. To me, it is very strange to read one strongly supports one form of privacy, while totally ignoring other forms. In fact it seems to me these operators are not only ignoring other forms, but it seems they do not even acknowledge the fact that to *some* people in the world the other (GDPR) form may be very important as well. Remember, people in different parts of the world do have different values and different needs. Arnold On 15-08-2019 18:39, Robert J. Hansen wrote: >> Well, it was just one of many example sites... > > Again: I'm going to go with the real advice given to me by real lawyers. > >> So as an example, US SKS key server operators do not have to honor >> removal request (in this case shut-down the server) from EU citizens, >> when they receive a letter from a lawyer? > > Depends on the individual. I rarely travel to Europe and have no > financial holdings there. It gives me a great ability to say "no, I'm > not signatory to your treaty, go away." Other Americans may have enough > ties to Europe to make it possible for EU courts to apply leverage. > >> I remember also that plenty of US sites (small and large), where I >> did business with, asked for my consent as EU citizen, when they >> changed their privacy policy once the GDPR took place. > > Some of them do business in Europe and are susceptible to pressure by > the EU. Some of them were just jumping on the bandwagon. > >> Has an US SKS key server operator then not 'business' ties with EU >> citizens, when storing their personal data like name and email address? > > No. Those are considered facts no different than tracking a name and > phone number. Mere facts cannot be suppressed by the United States > government; citizens are allowed to share them to our heart's content. > >> And has Mr. Rude then the right to freely distribute this data, without >> protecting it, to the whole world? > > I don't know anything about him or where he lives or which laws he must > follow. > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel > ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Implications of GDPR
On 04-05-18 10:44, dirk astrath wrote: > It may make sense to keep a revoked key in our database: > How is your decision, if a key can't be found on a keyserver? > Trust on first use? Don't trust? This should, IMO, all be a local, individual decision. I am currently looking into 'value sensitive design', making systems that are adjustable to specific human values in different human societies. [ https://vsdesign.org/ ] > If my key gets compromised, I want ... > Therefore: > If ..., I'm unable to ... In many messages in the current discussion, it is very easy to spot examples of a need for value sensitive design, like in these few lines above. > All in all: > > It's not easy to find a perfect solution (if there is any) ... and it's > not the first time we have this discussion ... In 2015, the idea came up to let go of the idea of a single set that gets synchronised. Instead of one single set, we can define parametrised filters to create 'value sensitive sets' ;-) [ http://lists.nongnu.org/archive/html/sks-devel/2015-05/msg00062.html ] If keyserver A has configured Set A and keyserver B has configured Set B, the set reconciliation algorithm between keyservers A and B should only consider Set C, where Set C is the intersection of sets A and B. The set reconciliation algorithm can be used the same. In case the operator of keyserver A finds that his Set A is "larger" than the union of all Sets of its peers, that operator can ask on this very mailing list for peering with keyservers that have the missing subset (based on the configured parameters of the filters). At least this might be a solution to the keyserver compatibility problem. For the pool, we might end up with sub-pools based on filter parameters. If operators don't do fancy filtering of their own, but follow local law, we might end up with regional pools, possibly based on groups of countries, with a maximum of the number of countries in the world ;-) Kind regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Oh, Jeeez...!
On 24-05-16 18:17, Tobias Frei wrote: > Adding proof of work can only prevent an attack that depends on a huge number > of > useless keys. Setting a maximum upload size can help and is easy to implement locally. Further, it is possible to limit the rate at which a single IP (or IPv6/64) can upload new or updated keys. > Someone else once mentioned that a single key with an illegal image > ID can already cause huge problems, and deleting such a key can become the > only way > to be legally allowed to continue running a keyserver. A possible solution I suggested a while ago is to have SKS synchronize only a subset of the database. The Set Theory that SKS uses, can be applied to subsets all the same. All we need is a good definition of the subset(s). If we agree to synchronise only the keys modified in the last three months or so, then everyone has the option to locally(!) delete keys that have not been modified for some time. Right now we ask new operators to have a recent dump before we start peering anyway. Yaron thought something like this is "totally doable" (see his e-mail of 21 May 2015 21:16:36 +), but it needs work... > About lacking keys, well, if the pool selection mechanism causes working > keyservers > to be removed, that's a separate problem that needs to be solved after this > one, I > think. It should not be an argument for or against this suggestion, but > instead > needs to adapt to the current situation. I guess the stats of SKS can be modified to show more stats, like the number of keys in the subset that is being synchronised. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Seemingly corrupted DB, not syncing
On 06-08-15 17:50, Gunnar Wolf wrote: Daniel Kahn Gillmor dijo [Wed, Aug 05, 2015 at 07:22:26PM -0400]: So... In your experience, what should be done? Is my best bet to just drop my DB and download a set of dumps again? You might be able to just drop your PTree (not the DB) and have sks rebuild it, without needing a new set of dumps. I'm sorry to not have any more sophisticated suggestions. bdb's failure modes continue to perplex me. :/ OK, this *seems* to have worked: After removing and creating a new, *empty* /var/lib/sks/PTree directory, I started sks, You did run something like $ if ! /usr/sbin/sks pbuild -cache 20 -ptree_cache 70; then echo fail; else echo OK; fi; tail /var/log/sks/pbuild.log to rebuild the PTree database, did you? I would be surprised if it works without (but it might just as well do :-) ). As a last resort, instead of downloading a full set of keys, you can stop all sks processes and generate a key dump yourself. It only needs the DB for that (PTree is for the recon process). Good luck! Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Monit and Munin script for sks server
On 19-07-15 17:58 +0200, Kristian Fiskerstrand wrote: Looking at https://sks-keyservers.net/status/ I see These statistics were last updated: 2015-07-19 19:35 (UTC) Kristian, did you update something on the monitoring that did not turn out as expected? ;-) Kind regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Key subsets
On 21-05-15 23:16, Yaron Minsky wrote: This seems like a very nice idea. There are some algorithmic challenges, though. Thanks! No challenges, no fun ;-) Right now, SKS maintains an incrementally updated data structure corresponding to a kind of magic checksum of different subsets of keys. These magic checksums are computed on subsets determined by a pre-arranged partitioning of the space. I guess this is the P of the PTree database? To do the reconciliation you propose, you'd need to be able to instead get your hands on the checksum of every key in a given range that passes a given filter criterion. If we have a known set of filters, well, then we can compute those incrementally in advance. If the filters are small (e.g., ban the following 30 keys) we can do them on demand. But if you have filters that are not known in advance but make big changes to the set of admissable keys (e.g., drop all keys minted before the following date), well, that's trickier. It might help that the set of filters you have to apply are (more or less) fixed for each of your peers. It might also be possible to determine the (most likely large) set of keys that all your peers want, and do those computations in advance. Or, perhaps, the total space can be divided in several subsets to cover all combinations you need for your set of peers. For each subset the computations can be done in advance and incrementally. The 'ban list' filters will usually contain a (relative) small number of keys. The number of filters having larger impact might be limited, I guess. If we only allow a limited set of parameters or no parameters at all for things like 'age' filters that impact is even more predictable. Another approach might be to limit 'normal' reconciliation to all modifications of the past n months (a limited number of keys, easy to calculate on demand). Once every x weeks you can do a 'full' reconciliation. Or, perhaps better, once an hour you do a 'historic' reconciliation, each time going one month further back in time, or pick a random month in history from now to -20 yrs. After all, we expect operators to start with a fresh dump. Changes along these lines are totally doable, but would require someone to really dig seriously into the SKS codebase. I'm not going to have the time to do it myself. I'd be happy to assist in thinking about the architecture and the algorithm, etc. However, ocaml is one of the languages I don't speak. I guess it won't be much trouble learning to read it, but I don't have the time to really learn to write. Arnold y On Thu, May 21, 2015 at 3:29 PM Arnold sk...@mallos.nl mailto:sk...@mallos.nl wrote: On 20-05-15 00:18, Yaron Minsky wrote: Let's think about a simpler question: deletion. Hmm, simple? This question alone has at least two flavours: A) local deletion B) global, network wide deletion Can SKS support outright deletion of keys? I think a fundamental question is one of trust: is there a trusted set of people who could do the deletions? Perhaps it's better to forget about deletion... It is no longer necessary once we leave the idea of a single set of keys :-) If so, then one could pretty easily add the notion of a deletion certificate that could be gossiped around like a key, and would essentially eat some set of keys, causing them to be deleted from the database. The problem is, I don't know that such a universally trusted set exists. So, what do we do? We could imagine having people volunteer to manage sets of banned keys. It came up before, multiple operators want to ban different kind of keys for different reasons. There is no single set of banned keys. You could subscribe to a banned key service, and just always reject any banned keys from your store. The question then is, what happens during reconciliation? Maybe at reconciliation time, everyone just pretends to have all of the banned keys in their store, and so no one ever detects a difference based on banned keys. I haven't really thought much about the literature in this area for a few years. If real progress is to be made here, someone has to think carefully about what would be an effective protocol. To think about SKS, you don't really have to know anything about the fancy math behind reconciliation. Just think of it this way: you need to represent your knowledge as a set, and SKS gives you a cheap way to discover the symmetric difference of your set and someone else's set, and then you can synchronize based on that knowledge. Deletion certificates just act as another thing in the set, and so everything works pretty straightforwardly from there. If you want to design a better SKS, you should think about
Re: [Sks-devel] Proposal: Start verifying self-signatures
On 19-05-15 13:25, Gabor Kiss wrote: I really like the idea of only accepting self-signed stuff as it would raise the bar for vandalism. No one is kept from generate a million of new regular looking self signed keys with some additional unwanted content. No, but it could be the first small step to our final goal of control over the content that we store (and that we make available for download to the public). Best regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Proposal: Start verifying self-signatures
On 18-05-15 01:37, Robert J. Hansen wrote: This is a DOS because Mallory could effectively increase Alice's public key to a size that it would be untenable for Bob to download it from the pool. There are so many other, better ways to DoS the entire keyserver network that I have real trouble taking this one seriously. It amazes me that each time something comes up that involves control over the content of our database (cleaning, removing, rejecting at the gate, etc.), discussions quickly focus on unimportant details leading to the conclusion: let's do nothing: accept everything and keep everything. I tried, but really could not think of an example of the opposite in the past six years. I guess the real problem for further key-server development is there is no common vision or goal for the SKS-network. I really doubt it is possible we ever agree on one (or even multiple) either ;-) At the same time I strongly believe accepting and keeping everything is not future proof: the SKS-network _will_ collapse sooner or later. Reading the archive of this list provides many options... The main question is: do we (want to) let that happen? Regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Proposal: Start verifying self-signatures
On 18-05-15 21:26, Johan van Selst wrote: Daniel Roesler wrote: Uploading user attribute packets with bogus self-signatures is probably the easiest way to DoS the entire keyserver network. A bot could add 1TB of bloat to the keyserver network by adding 5MB (to stay under the limit) user attribute images to only 200k public keys. By contrast, assuming a signature is 2KB, they would need to submit 200m bogus signatures to have the same impact. Then again, generating a batch of bogus signatures is a rather trivial task as well. So Johan, do you mean let's do nothing, as this single one proposal does not protect us from *all* evil? Or do you mean we have to implement the proposal and think about how we can mitigate other attack vectors to the SKS-network, like the one you mentioned? Regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] sks.disunitedstates.com down and out
Hi David, On 11-04-15 12:15, David Benfell wrote: Quoting Hendrik Grewe hendrik.gr...@tu-dortmund.de: Also what I have read about your tries to recover the database. I think I have read in some FAQ that one should _not_ try to recover a (somehow) corrupted sks database. Instead one should start (from skratch) with some actual keydump. Which by itself is a very strange recommendation, however true. Among the steps I took was indeed a recovery from an actual keydump. I have just seen way too many problems with Berkeley DB in way too many situations. After I had a few times problems, I changed strategy: every 2 or 3 months I generate a new dump, throw away both databases (do s/w upgrades if available) and do a fresh 'fastbuild'. In case of BDB problems in between I do the same, although I might re-use my previous dump. SKS is definitely taking more administrator time than any other service I've run. Best regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Potential problems running keyservers
Hi, On 26-10-14 10:17, Hanno Böck wrote: From my understanding the core principle of the pgp keyservers is that they have an add only-policy, meaning you can never remove something, just add further information to it (e.g. keys don't get removed, they expire or are revoked). Correct. This opens up a couple of problems and I wonder if they have been discussed before and if there are any counterstrategies to them. a) Someone could just flood the keyservers with random bogus keys. This would basically fill up the hard drives of the keyservers. b) Someone could grow a target's key by adding more and more signatures. This would quickly make downloading the key from the keyservers infeasible. c) Someone could use keys, keyids, signatures or whatever to store illegal data. (Basically this very same issue has already been discussed in the context of bitcoin [1]) These things have been identified in the past as 'possible issues'. Some people on this list, like myself, would see measures implemented now to be able to counter these issues at the moment they get real. Others argue nothing has to be done at this moment. Currently, there are no developments or even strategies to counter issues like these. d) A fourth possible issue is the fact we store personal data. Many countries in Europe have laws regarding the storage of personal data, including the right to get removed. Up to now, we have had one server operator who had to close down due to legal threats regarding this. I don't really see any feasible counterstrategies to these issues. Situations c) and d) can be countered by making it impossible to retrieve the data of keys that are listed in a local list that the server operator maintains. This keys should not be listed in search results either. If getting keys by SKS-hashes is excludeded from this mechanism, it would still allow SKS-sync as normal. b) can be countered by setting a max. key-size servers accept. In this case a server has to reject new material in case the resulting key is larger than a certain size that has to be agreed among peering SKS-operators. To avoid loosing revocation certificates, we could strip revoked keys from all unnecessary information. a) is difficult... One could restrict the uploads from a certain IP, but a botnet uploading to every known keyserver can still do a lot of damage. Given the speed one can generate and upload material to key servers (keys don't have to be valid to be accepted) I think all three scenarios could easily happen. Yes, if someone wants to cripple relative easy key exchange, that is very well possible. Without a public key infrastructure like our SKS-network, use of OpenPGP for signing / encrypting e-mail, documents or server certificates (http://web.monkeysphere.info/) will be much harder for the average user / organisation. I'm curious what the thoughts of the people running keyservers are. My € 0,02. Arnold [1] https://www.reddit.com/r/Bitcoin/comments/1akyy4/what_happens_if_someone_inserts_illegal_content/ ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] ams.sks.heypete.com migrating to new facility, changing IP addresses
On 12-09-14 19:28, Pete Stephenson wrote: For those who are curious, the process went like this: Thanks for sharing this. One suggestion for improvement for those who want to do something similar. ... 8. Wait several hours. ... 9. When there's no SKS traffic to the old server, turn it off and enjoy a cold beer. Rename 8 into 8b. Split 9 into 9a and 9b and copy 9b as step 8a. Enjoy :-) Cheers! Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Broken keyservers (413 Request Entity Too Large)
On 04-09-14 08:16, Christoph Egger wrote: Seems uploading my gpg key (d49ae731) to pool.sks-keyservers.net fails for several of the hosts in rotation: The question is: is the key too large, or should we accept keys of *every* size? Accepting every key size does not scale well in the long term. It can also lead to a nasty DOS attack: upload many huge keys to eat all the public key server resources. We currently have no means to remove keys or specific key data. People with a very large key can put their full key at a special place of their own (they are likely to be above average active internet users). They can still upload their key with exp. time and all textual UIDs. However, they should remove most of the signatures and picture UIDs and instead include a 'preferred key server' field. That way, the pool helps to *find* and distribute the key with a few signatures. People receiving the key should then update that key, which will automatically use the preferred key server. This also brings us back to the question: what is the main purpose of the pool? - Is it the ultimate key archive including all UIDs and (expired) signatures? - Is it the place where you find all info you need to compute your trust in a key (i.e. revoked and expired keys are stripped down and expired sigs and pictures are removed)? - Is it a central place to find public key data (we focus on storage of key material, expiry and revocation data and textual UIDs)? - ... The problem to answer this last question is that we all operate an SKS-key server for our own reasons... Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Broken keyservers (413 Request Entity Too Large)
On 04-09-14 22:13, Kim Minh Kaplan wrote: David Benfell wrote: On Thu, Sep 04, 2014 at 02:31:38PM +0200, Arnold wrote: On 04-09-14 08:16, Christoph Egger wrote: Seems uploading my gpg key (d49ae731) to pool.sks-keyservers.net fails for several of the hosts in rotation: The question is: is the key too large, or should we accept keys of *every* size? Accepting every key size does not scale well in the long term. It can also lead to a nasty DOS attack: upload many huge keys to eat all the public key server resources. We currently have no means to remove keys or specific key data. Actually, I think this isn't the problem you're making it out to be. Ellyptic Curve Cryptography keys are much smaller and will be supported in GPG 2.1. Some implementations of 2.0 also seem to support these keys currently. The largest RSA key size I've seen implemented is 8192. This is in APG (the Android variant). I would suggest setting an upper bound of 16384. Obviously Arnold is not referring to the cryptographic key size but to the complete OpenPGP key size, the whole shebang. 0xd49ae731 has many uids each signed with loads of signatures. It is close to one million bytes in its armored form. Indeed. Still I do not see how limiting the size of a single key would protect the SKS key servers from a DOS. To an attacker uploading many huge keys has about the same difficulty as uploading many many big keys. Yes, that is true. However, not limiting the key size (or taking effort to raise the limit that seems to be in effect) will not help in any way. With a limited key size as first step, it is easier to implement further measures like limiting the amount of keys one can upload per hour from a given IP subnet (IPv6 proof ;-) ) Fortunately, currently we suffer no attacks, but (like many open standards and open software) we take good intentions for granted. It won't harm us to think about the weaknesses of our system and then decide (educated) if we take the risk or take measures. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Broken keyservers (413 Request Entity Too Large)
On 04-09-14 22:26, David Benfell wrote: Even worse, then. I don't see this use as an abuse, but as legitimate. We should be able to accommodate it, even if it is an outlier. You mean no limit at all? RFC 4880 has no limit, so a key of size 1+ Tbyte can be totally valid. It might even make sense to have such key for a certain purpose. However, I see no reason why we should accommodate distribution of special (large) keys as a free community service. On average keys are only about 2 kbyte on disk. Idealistic, we should go for 100% of the keys and trust the good intentions of the people uploading keys. But, we all know there are people with bad intentions too. Practically, rejecting some keys seems reasonable to me. Especially if we can prevent DOS and abuse of our servers that way. A large key might as well contain a collection of HD pictures of a certain kind. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] status page
Hi list, Trying to be a bit more on-topic again... On 04/18/2014 10:42 PM, Simon Lange wrote: Am 18.04.2014 22:10, schrieb Martin Papik: Answering ALL host names just makes you willing to participate in any pool by default, without extra maintenance. But again, AFAIK this isn't a requirement. Am I misinformed? https://twitter.com/krifisk/status/456717051340791808 With a HTTP Host header not belonging to the specific hostname? Note the -H 'Host.' , 11371 should allow ALL traffic through I figured this is because of the following. HTTP is a unique protocol in that it allows the client to provide the hostname of the host it is communicating to (basically it is a selection parameter a client can provide, regardless of the hostname it resolved, for servers hosting several websites). Other protocols, like SMTP, NTP, etc. just listen at a given IP/port that the client 'somehow' manages to obtain. THAT is how most of the internet works. I have not looked into it but I am not even sure HTTP requires the 'HOST' header field to be present in requests. AFAIK it was not sent in the early days. The key exchange protocol HKP is not an RFC standard. AFAIK it is best specified in http://tools.ietf.org/html/draft-shaw-openpgp-hkp-00 It does not mention the 'HOST' field, so I guess openGPG implementations using HKP are not required to provide a (valid) HOST. Note that HKP is a protocol by itself (P in HKP), just reusing some of HTTP. People implementing HKP for small devices (Android, iOS, embedded systems, etc.) may omit sending the HOST, although they may be using our pool domain. Therefore, we (people in the pool) can make no assumptions on the value of the 'HOST' parameter. Therefore, it is good practice (if being part of the pool) to respond to all clients connecting to port 11371. And therefore, I don't provide the HKP-service at port 80 over IPv4. My choice :-) How would bad people benefit from your key-server responding to http://very.bad.com:11731/ anyway? bad ppl could pretend offering a public service using my machines they dont own nor they administre nor they run. my machines would support that passivly. think this is easy to understand. and also has some legal implications. No, although it might give you trouble explaining to legal people how DNS and the internet works... It is just not in your hands what IP's other people put in their DNS. People who associate you with the owners of 'unwanted domain' simply don't understand the internet. I understand there might be situations that can make you feel uncomfortable, especially when serving a website. However, among the millions of keys we serve, I am sure there are keys of people that I don't want to be associated with. Therefore, I try to be associated with 'undiscriminating' key server administrators and hope this outweighs possible negative associations :-) To me, it is even better if Mr(s) Bad points his/her own domain/DNS to my key server to get his/her key, than prominently being listed with my own domain name on the website of Mr(s) Bad. That would even result in a google-association... Off-topic As a side note: curious how easily some discussions on this list go into extremes, followed by getting personal and people getting hurt. Usually that kind of emotions is a sign of dedication to the subject, making it even more painful... As I am open to every client connecting (though rate limited), I am also open for peering with every SKS operator. Just send me your info off-list and I'll send you mine. Just my € 0,02 Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] status page
Hi, On 04/17/2014 04:20 PM, Simon Lange (BIT) wrote: well, but there IS a reverse proxy. ;) tcp0 0 78.46.21.218:11371 0.0.0.0:* LISTEN 8804/lighttpd If I remember right, the monitor checks for a 'VIA' header. See https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering for the configuration of several proxe servers. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] How much load are keyservers willing to handle?
Hi, On 12/19/2013 02:04 AM, Jason Harris wrote: On Wed, Dec 18, 2013 at 10:20:26PM +, adrelanos wrote: Writing that script would be much simpler if it could re-use the existing keyserver infrastructure. Now imagine if this gets added to Debian, that all users of Debian and all its derivatives will always refresh their signing key against keyservers? Could keyservers cope up with the load? 1) setup your own DNS so you can shut things off if anything goes wrong! (you can use dyn.com or others, no servers required) Analogue to the NTP pool [1] I would suggest to create special pools within sks-keyservers.net for vendors, like vendor.pool.sks-keyservers.net. This pool will be equal to the main pool in normal circumstances, but can be modified in case of problems. That way we stay in control. [1] http://www.pool.ntp.org/en/vendors.html Best regards, Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
[Sks-devel] Strange reason in status page
Hi, Some maintenance on my server is taking a bit longer than anticipated. While waiting for some command to finish, I checked the sks status page for my server. The status is Not OK, so that's OK. Next to look at was the reason: Reason Not running a reverse proxy However, the reverse proxy is the only thing that _is_ actually running :-D OK, my server is up and running normal again. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] IPv4 vs. IPv6? -- Reconciliation attempt from unauthorized host, but host is authorized
Hi Daniel, On 11/27/2013 06:57 PM, Daniel Kahn Gillmor wrote: i'm running sks 1.1.4 on Debian GNU/Linux, wheezy, amd64 (x86_64) same, but sks 1.1.3, on a virtual machine (kvm/qemu). Can anyone with a dual-stack machine (both IPv6 and IPv4) verify a successful connection from an IPv4-only peer in their recon logs? Don't know if they are IPv4-only ;-) My system is connected to (native) IPv6 (global address) and IPv4 (via NAT). Here are some details of my settings and from my log. I guess you have in sksconf something like :: or 0.0.0.0 instead of explicit addresses, especially for IPv6. I found some remarks (of myself) in my conf file to use explicit addresses for IPv6. # cat /proc/sys/net/ipv6/bindv6only 0 # netstat -l tcp0 0 pgpkeys.mallos.nl:11370 *:* LISTEN tcp0 0 localhost:hkp *:* LISTEN tcp0 0 pgpkeys.mallos.nl:hkp *:* LISTEN ... tcp6 0 0 pgpkeys.mallos.nl:11370 [::]:* LISTEN tcp6 0 0 localhost:hkp [::]:* LISTEN tcp6 0 0 pgpkeys.mallos.nl:hkp [::]:* LISTEN # cat /etc/sks/sksconf | grep _address recon_address: 192.168.178.50 2001:980:53c0:1:46a:efff:fecf:701b hkp_address: 127.0.0.1 ::1 # less recon.log Beginning recon as server, client: ADDR_INET [209.62.30.226]:51745 ... Beginning recon as server, client: ADDR_INET [2a00:1280:8000:2:1:8:0:1]:57649 ... Recon partner: ADDR_INET [209.62.30.226]:11370 ... Recon partner: ADDR_INET [2a00:1280:8000:2:1:8:0:1]:11370 Etc. Hope this helps! Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 11/01/2013 06:01 AM, Petru Ghita wrote (a few days in the past??): On 11/04/2013 10:14 AM, Johan van Selst wrote: Petru Ghita wrote: But I don't really think that such a legal action is possible and assuming it was possible that it would have any degree of success. [..] To sum it up: - there is by architecture no intent on verifing nor identifying the information stored on the SKS network nor the author of the data. It doesn't matter if the information is verified. Users are asked for their name and email address, which is considered personal data (according to EU definitions) and keyservers are processing and storing this data. Thereby, keyserver operators are subjected to the data protection laws. The validity of the data is not relevant, neither is the intention of the operators (commercial or otherwise). Users are not asked for their name nor for their email address by the SKS implementation. Please check [1]. What I see there is a text field, that has no validation nor any standard format whatsoever that is used to identify a public key. True, but the point is I have data in my database that I make publicly available. It does not really matter where it came from or who put it there. Someone may have objections to the publication of that data. If a judge supports the objection, I have a problem. This is the main argument any SKS operator has in front of a judge, in No, not for me. I make available data that the owner wants to be publicly available. I do this _in good faith_ people uploading a key do not abuse the UID fields (or other fields) of OpenPGP keys. However, if abuse of UID fields is brought to my attention, a judge in NL will not accept that I take no action. There are multiple laws in NL on which deletion of data can be requested. Privacy law is just one of them. Other examples are child porn in a photo ID, insulting people or damaging one's good reputation, copyright protected data of a book (a UID can be very large), etc. These are just some things I can think of. my opinion. The whole point being that there is no such thing as personal data stored in the UID. The definition used in your local law may be different. ... What I'm trying to show here is that I think there is quite strong evidence that a SKS server or the SKS network for that matter is just a storage and delivery media, same as a distributed web server or a web proxy cache. If you are fine without the possibility for deleting a key, that's fine with me. However, I can think of situations that I definitely will have a problem. Therefore, I welcome the possibility to delete (or at least hide) specific data, without the need to stop the service completely (_if_ the situation occurs). I just hope the ones who are fine without the possibility to delete or hide data support the ones who feel the need to have it. We can then start the discussion what needs to be done. It is also fine with me if it is decided (idealistic) that it is more acceptable to have key servers shut down under legal threat (or by abusive large keys), than to have the possibility to delete or filter. Up to now, I only read we have to be careful with deletion as there too is the possibility of abuse, harming some unknown user who fully relies upon our key server network not deleting anything. Currently we still seem to be in the state of debate whether it is valid or not that some feel they might need to delete or hide keys. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 11/04/2013 10:14 AM, Johan van Selst wrote: Petru Ghita wrote: To sum it up: - there is by architecture no intent on verifing nor identifying the information stored on the SKS network nor the author of the data. It doesn't matter if the information is verified. Users are asked for Not only are they asked, they can provide whatever they like, possibly with the intention to harm ones reputation or cause damage to a person in some way. their name and email address, which is considered personal data (according to EU definitions) and keyservers are processing and storing this data. Thereby, keyserver operators are subjected to the data protection laws. The validity of the data is not relevant, neither is the intention of the operators (commercial or otherwise). I think (I am no lawyer) the law (in the Netherlands) aims at organisations, not individual citizens (you can have a private address book). However, I don't want to find out in court. If national or international data protection laws give users the right to have their personal data removed from servers, then it should be possible for local keyserver operators to comply with that law. Preferably without terminating their service. I second that. The privacy and data protection regulations are not the only thing to worry about. If people put Nazi slogans or death threats into UID fields, or put child pornography into JPEG attachments, then there may be other laws that can force keyserver operators to remove keys. True. As SKS operator I am responsible for the data on my system. The fact I *choose* a software system (SKS) that is 'all or nothing' does not prevent a judge from telling me to remove particular data. If that means I have to remove all data, then that is my problem. IMHO there is a clear demand for the option to remove certain keys - or at least make them irretrievable locally. That some keyserver operators are asking for the feature, should be reason enough to move on to discussion of the technical aspects. We have a few questions that needs to be answered before going to technical solutions. These are some questions I can think of right now, please extend :-) Q1) is it sufficient to hide data for users of our service, while we keep full key data in our local database? Q1.1) if we hide data for the users of our service, do we hide all (like the key does not exist), do we strip uid's, but still return key material including key expiration, revocation, etc.? Q1.1.1) if we hide, but still provide some data, is it retrievable by key id only, or also as search result on name or e-mail? Q1.2) if we keep full key data in our database, are we allowed to actively send that data (push or pull) to our SKS peers? Q1.3) if we keep full key data in our database, do we allow that specific key to be updated by users of our service (i.e. from GnuPG, web interface)? Q2) if we remove data from our local database, do we remove all data of a key, or do we just strip some data (at least all uid's)? Q2.1) if we strip some data and keep some other, which data do we strip and which do we keep? Q2.2) If we keep some data in our local key server, but strip for example uid's, do we still update that key with other data (like expiration date of the key or a revoke certificate)? Q2.3) If we remove some or all data of a key, do we inform our peers during synchronisation? Q2.3.1) what to do when we receive a 'delete indication' from a peer, do we want to get a warning, do we want to delete the data also from our own server, do we want to delete that data only if we receive it from a peer we have marked as 'trusted' in our membership file, something else? Once we know what we want (ideally), the OCAML developers can see what is technically feasible short term, long term, 'at all' and which things are configurable (like only hiding, stripping uid's, totally removing a key, any mix per key, etc.) If the technical solution results in some servers providing data other servers do not, then we will have a follow-up discussion about which servers to include in the pool. But let's first get clear what we want as individual SKS operator. Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 10/29/2013 11:15 PM, Kristian Fiskerstrand wrote: On 10/29/2013 11:04 PM, dirk astrath wrote: If there is no private key needed and no verification done everybody can generate keys with every combination of name and email-adress, generated at random dates and upload them to the keyservers. And if everybody is able to generate and publish fake keys everybody can build up fake web of trust. This is why you have key validation requirements and signatures/certification. The existence of a key doesn't bind that key to a specific individual, no matter what the UID says. Wrong ... the unique email-adress is the problem .. which is usually in the UID of the key. This isn't too relevant from a security perspective wrt a fake web of trust but seems more like a response wrt privacy questions. But, privacy questions are the thing this is all about. I don't expect anywhere to be a lawsuit against a key server operator for providing keys without trusted signature on a UID. However, we already had an example of a key server being shut down because of legal threat based on illegally providing personal identifiable data (according to some local law). Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 10/29/2013 11:40 AM, Kiss Gabor (Bitman) wrote: Several times in the past years the problem of deleting keys on user request is discussed. ... The fundamental problem was that some users want their keys to delete from _all_ key servers. What users _want_ is not very relevant (well, sort of...) but more what they legally can enforce you to do. They can (threaten to) enforce specific key server operators to remove a key from their key server (not from the network). As we have seen already this is not possible for technology reasons. If key removal is useful or desirable is a totally different question. Agreed, with the legal threat the free open key server service we provide is at stake. Some guys argue against it saying it would make impossible to check digital signatures made by the deleted key. I reply: who cares? The situation is the same if the user never upload his/her public key to a key server. Also the same if there are no key servers at all (all threatened and out of business). The problem is have to solve -- I think -- that users threaten key server operators with legal actions. I have a proposal that may a trade-off. IMHO most of the complaining users will accept that their keys remains in the database but they are not appear in search results. Providing less technical detail to those users is better ;-) Why tell it remains in the database, just agree to 'make it unavailable from your specific key server'. From what I know from our national law on privacy, operating a key server is OK, but... if a judge decides otherwise, keeping it in the database won't be acceptable either. Therefore, I agree to have an easy to implement solution (for the key server operator) that a) makes it possible to react on requests instantly (before legal threats and lawyers etc.), and as a consequence b) keeps a key server in business. Technical implementation is the following: If a user wants to hide his/her key (s)he just have to add a special uid e.g. Do not include in search results or so. If I remember right, there was a situation that Alice created a key with the name of Bob. Bob complained to the key server operator, but he is not able to modify the key Alice created. So, the key server operator should be the one who disables retrieval of the key. A technical solution I think of is to limit the search engine based on a list of (primary) key ID's (full fingerprint). That list is a local configuration file. For the key server network to function, I guess it is still necessary to provide the key based on key hash. Currently it is possible to show the key hash in the search results of the web interface. I think it would be wise to hide that hash as well. Otherwise, Bob can still argue that the key server operator still provides his key. And Bob would be right IMO. Does showing the hash serve any practical use? If it is only useful to SKS developers and/or operators for testing and debugging purposes, it might be an idea to make the hash only available via a debug option that is normally disabled. The search engine just should ignore these keys. However key could be retrieved by hex keyID that makes That won't help a key server operator who was legally forced not to provide the key of Bob any more. ... Yes. Very smart and desperate end users and their lawyers may point out that the key is actually NOT deleted, and an impostor If we don't tell... ;-) Only people digging into this mailing list and people reading the source code may know. But *if* a lawyer knows, I guess he will go for it (he is paid by the hour). It is not their concern if you have no other option than to stop the service at all. BTW. The dumper could also drop these keys. Also the recon process. Or does this consume too much resources? I also once played with the thought to only store the hash and key fingerprint in the database to satisfy database equality. But, once you update the hash from server A, then server B (not peering with A and then with an outdated hash), will keep requesting the key associated with the new hash that you cannot provide. This will continue, unless server B also replaces the outdated key with the new hash only, effectively also deleting the key which we concluded before is undesirable... Just my €0,02 as I am no SKS developer. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 10/29/2013 03:51 PM, Kristian Fiskerstrand wrote: On 10/29/2013 03:47 PM, Arnold wrote: On 10/29/2013 02:30 PM, Kristian Fiskerstrand wrote: The discussion gets even more interesting when dealing with revoked keys. If an attacker (with compromised secret key material) is given the ability to deleting such a key from the server network and re-uploading a non-revoked version; The effective security of the whole system is compromised (or for that matter mallicious key server operators doing the same). There are good reasons for the servers being add-only by design... and you'll find several discussions on this in the past. True, the SKS *network* should be add-only by design. However, this does not imply that each key server is required to _provide_ each and every key in its database or in the SKS network by means of a search. If the SKS network (currently) can operate based on search by hash only (which I assume) and if no current practical use (other than test or debug) is hindered by hiding the hash from search results (disabling the option hash=on), then filtering search results by means of a list of key fingerprints seems feasible to me. I don't understand your point here, can you please elaborate? For clients accessing the pool the results are simply a DNS round robin and the client connects to a given SKS server. Good you mention the pool. From my point of view the or a SKS network is just a number of individual key servers running on individual hosts, operated by individual operators/administrators. The operators have little agreements with a few other operators to peer with each other and thus form a network. From this point of view our pool is just one SKS network. Multiple may exist in the world. Our network provides additional services besides offering keys and synchronising among many servers. We provide a single 'server host name' (DNS) that resolves to multiple key servers with some kind of quality in terms of availability and completeness of the database. I deliberately separate the two. First thing is the possibility to hide / remove / whatever a key from a network of individual and independent key servers (operating under different local laws). If that is sorted out, we can discuss how to continue to provide the additional services like the single DNS for the whole network and maintain our own lever of (internal) quality. Hope this explains my point. If there is fragmentation in the network we'd have to split the servers (probably exclude servers Excluding servers might not be scalable well as more countries implement some kind of privacy law. Just a thought, but this should really be step two. The DNS servers of the NTP-pool (http://www.pool.ntp.org) dynamically provide 'nearby' pool servers. We might do the opposite and provide 'far away' servers as it is unlikely those servers are forced to remove 'nearby' keys. with deleted keys). My suggestion was not to delete a key, but merely to hide it in search results (to the highest degree possible). That highest degree possible is IMO a must to prevent lawyers actually demanding (and legally enforcing) deletion of the database. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] About deleting keys
On 10/29/2013 11:25 PM, Kristian Fiskerstrand wrote: On 10/29/2013 11:05 PM, Arnold wrote: I deliberately separate the two. First thing is the possibility to hide / remove / whatever a key from a network of individual and independent key servers (operating under different local laws). If that is sorted out, we can discuss how to continue to provide the additional services like the single DNS for the whole network and maintain our own lever of (internal) quality. If there is fragmentation in the network we'd have to split the servers (probably exclude servers Excluding servers might not be scalable well as more countries implement some kind of privacy law. The pool only require one single load-balanced server in the front located in some country where such limitations doesn't apply to fulfil its goal, synchronizing with the other servers to get key updates (although that certainly would be a suboptimal solution with regards to latency (in particular network roundtrips)). The scalability I was talking about was about the existence of multiple servers in multiple countries (to have available for balancing the load for one thing). If we don't take care of this thread, the SKS network might very well be reduced to a few servers in a single country very soon... I am not looking forward to rely on SKS key servers in some country where they log and analyse which keys I retrieve. Therefore, I split the two: first implement something so operators of SKS key servers can deal with local requests for deletion. Hiding may be phase one. It might be necessary to follow up with a phase two: actual local database deletion with some means in place to prevent resynchronising that key. Arnold ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] why does SKS have /dev/random open for writing?
My Debian system looks normal (only Debian stable/wheezy packages). lsof /dev/random COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sks 2638 debian-sks3r CHR1,8 0t0 1209 /dev/random sks 2639 debian-sks3r CHR1,8 0t0 1209 /dev/random uname -a Linux pgpkeys 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux dpkg-query -l sks ... NameVersionArchitecture Description +++-===-==-- ii sks 1.1.3-2amd64Synchronizing OpenPGP Key Server Arnold On 09/19/2013 07:31 PM, Daniel Kahn Gillmor wrote: hi SKS folks-- I was just looking at the behavior of sks 1.1.4, and i noticed that it seems to have /dev/random open for writing: 0 zimmermann:~# lsof /dev/random COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sks 742 debian-sks3w CHR1,8 0t0 1244 /dev/random sks 756 debian-sks3w CHR1,8 0t0 1244 /dev/random 0 zimmermann:~# this is not read/write (which would be marked as 3u instead of 3w), but write-only (presumably appending) to the character device. I'm not clear on why this is happening. I don't see /dev/random referenced explicitly in the source. Anyone have any clue about what it's doing? This is happening for me on debian systems -- can users of other systems confirm or deny that this is happening for them as well? Maybe it's an artifact of one of the ocaml libraries sks depends on? I'm not sure how i'd debug that to verify it. --dkg ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Requesting your thoughts on a web of trust scheme
On 12/09/2011 06:36 AM, Daniel Kahn Gillmor wrote: I'm not sure this is OT on sks-devel I think it is, as the proposal says: 'The proposal of the present document is to use PGP keys ... This will allow the trust data storage to piggyback on the existing network of SKS keyservers, achieving instant distributed storage mechanism with minimal investment in infrastructure' In other words, the proposal suggests to make use of resources donated by the community for one cause, *to save investments* in realising a different cause, without asking our permission. This by itself is a very bad thing IMHO. (The question of Daniel F was about comments on the scheme, not about permission to use our network.) Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] key.ip6.li status
Hello Christian, On 05/23/2011 05:49 PM, Christian Felsing wrote: I am not sure if there is a config problem in my SKS, but key.ip6.li is not listed at http://sks-keyservers.net/status/. Your server is listed in http://sks.spodhuis.org/sks-peers This has nothing to do with the DNS-pool, but gives you at least an indication your server is recognised by some script as node in the keyserver network. Any hints ? How does sks-keyservers.net detect status ? No hints, but it has detected your server is one of my peers. http://sks-keyservers.net/status/info/pgpkeys.mallos.nl (Still it has not detected my IPv6 capability, which was enabled before.) I guess it will detect your server automatically in a couple of days. Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] sks-network
Hello Scott, If I am right, your first message to this list was just three days ago, requesting peers for a new server installation. IMHO, it is impolite to tell a community what rules they should adhere to, if you're welcomed to that community only three days ago. This is especially the case if you are touching subjects that have been discussed (long) before. If you don't feel comfortable in our small community of SKS server admins and with the rules we (seem to) adhere to, then don't bother and find other SKS server admins to peer with and set up your own network of SKS servers. Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] sks-keyserver Fedora Linux
Hi, On 04/25/2011 03:00 PM, Jeff Johnson wrote: On Apr 25, 2011, at 8:29 AM, Robert Hinson wrote: I tried running the pre compiled sks keyserver on a fedora 14 box. I do the sks build ./dump/*.pgp -n 10 -cache 100 It craps out after like 2 minutes. Anyone else experience this problem? Yes. The problem is in fact described though somewhat obscurely: You need to adjust the -n and -cache parameters. There's a third path to works is loading a truncated dump. There are approximately 200 files (depending on chosen per-dump-file-size) used in initializing a SKS server. Removing some of those files is another way to achieve a load in spite of difficulties with crap out related (apparently, I do not know) to the amount of memory resources available. I used this route (on a normal build instead of a fastbuild). I did the initial build with something like 10 files. Then I did a for-loop on the shell for (small groups of) the remaining files using 'sks merge ...' With this I also played with -n and -cache. The advantage is you are working with relative little data at a time, so it is faster to determine relative good values for -n and -cache by trial and error. Like in sks_build.sh, you end with a 'sks cleandb' and 'sks pbuild'. If the latter gives you also problems, then (if I remember right) you can also do a pbuild after the initial build. 'Merge' updates both databases, but will take more time. The overall time was longer than normal, but I was more in control. Please note that removing files from the dump in order to achieve an initial load WILL lead to a very large update (I've seen 24 hours needed to catch up) when you start peering. This is prevented by 'merge' of the remaining files. Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
[Sks-devel] Backup/restore DB (Re: Server won't start after copy of DB)
On 06/27/2010 08:28 PM, Jeff Johnson wrote: On Jun 27, 2010, at 2:25 PM, C.J. Adams-Collier wrote: The question now is, how to do a backup that can be restored to a new disk? Isn't that the question you just answered? :) Well, not quite ... Indeed, I had my old database in good working order available (I just had to mount the partition) to do the dump. From that I restored (= rebuilt) the new database. This is very different from recovery from backup after the database is destroyed. Procedures for backup of a Berkeley DB (with logs that are path/record sensitive) are described in the doco @oracle.com. An SKS keyserver database is no different. OK, I've looked it up and here it is for the archive. The procedures are described in chapter 5 of the doc's: http://www.oracle.com/technology/documentation /berkeley-db/db/gsg_txn/c/index.html Important steps: - Before backing up the files (offline), do a checkpoint (db_checkpoint command line utility). - Use catastrophic recovery when you are recovering your databases from a previously created backup to an empty directory (db_recover command line utility with the the -c option). Although I did run db_recover, I did not use the '-c' option. It would have saved me some time... Hope this helps others! :-) Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Server won't start after copy of DB
Hello Peter, C.J., On 06/23/2010 10:04 AM, Peter Pramberger wrote: Am Mi, 23.06.2010, 00:49, schrieb Arnold: I moved the database to an SSD hard disk and since that moment it immediately exits after it starts. Silly question, but have you done a db_recover in the new location? Not a silly question! ;-) I am not so familiar with all the db_xxx tools. (I did db_verify, though.) Now, I've done db_recover (without options), but the same result. Note that the 'old' server was stopped in a controlled way, it was no crash. I'm not sure if the region files somehow reference to the original database location (or maybe inodes)... Thanks, it must be the inodes... Both the old and the new path are /srv/pub/sks/ (with a sym-link from /var/lib/sks to that location). So, it's not the path. This raises the question, how does one restore a database to a new system after a crash? Now, I mounted the old partition somewhere else and sym-linked /srv/pub/sks to that new location. SKS is running happily again!!! On 06/23/2010 05:40 PM, C.J. Adams-Collier wrote: oh. I just thought of something: You will likely get this done more quickly if you grab the snapshot and start over. I am afraid you are right... Now that the server is running again, I can create my own dump instead of downloading it and rebuild from that. This will be some job for the weekend. If, in the meantime, anybody comes up with a solution to reuse an existing database on a new partition with new inodes, I'd be more than happy to learn about it! :-) Thanks for the help! Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
[Sks-devel] Server won't start after copy of DB
Hello, My Debian SKS-server was running very well, until this weekend. I moved the database to an SSD hard disk and since that moment it immediately exits after it starts. These are the last lines of my db.log and recon.log, with debug level 100 (same result at 10 or 9): $ tail -n 3 /var/log/sks/db.log 2010-06-22 22:06:00 Membership: ADDR_INET 76.185.38.113:11370(keyserver.gingerbear.net 11370 ), ADDR_INET 76.191.185.172:11370(www.mainframe.cx 11370 ), ADDR_INET 83.169.43.165:11370(keyserver.ccc-hanau.de 11370 ), ADDR_INET 195.111.98.30:11370(keys.niif.hu 11370 ), ADDR_INET 69.134.24.120:11370(keys.rpm5.org 11370 ), ADDR_INET 198.49.244.154:11370(keys.n3npq.net11370 ), ADDR_INET 85.113.243.78:11370(keyserver.nijkamp.net 11370 ), ADDR_INET 140.186.70.102:11370(keys.sugarlabs.org 11370 ), ADDR_INET 84.16.235.61:11370(keyserver.pki.scientia.net 11370 ), ADDR_INET 66.109.111.12:11370(keyserver.kjsl.org 11370 ), ADDR_INET 203.33.246.146:11370(keyserver.oeg.com.au 11370 ), ADDR_INET 208.84.222.190:11370(ice.mudshark.org 11370 ) 2010-06-22 22:06:00 Opening KeyDB database 2010-06-22 22:06:00 Shutting down database $ tail /var/log/sks/recon.log 2010-06-22 22:06:00 sks_recon, SKS version 1.1.0 2010-06-22 22:06:00 Copyright Yaron Minsky 2002-2003 2010-06-22 22:06:00 Licensed under GPL. See COPYING file for details 2010-06-22 22:06:00 recon port: 11370 2010-06-22 22:06:00 Opening PTree database 2010-06-22 22:06:00 Setting up PTree data structure 2010-06-22 22:06:00 PTree setup complete 2010-06-22 22:06:00 Initiating catchup 2010-06-22 22:06:00 Marshalling: LogQuery: (5000,1276993585.849555) 2010-06-22 22:06:00 DB closed The system is running - Debian Stable 'Lenny' with - Linux 2.6.32-bpo.4-686 (from the back ported archive) - SKS-server v 1.1.0 The database is now on this partition: - /dev/sdb13 on /srv type ext4 (rw,nosuid,nodev,user_xattr) With the change of HD, I also moved from ext3 to ext4, which is better for an SSD. Ext4 is also the reason to use the newer, back ported Linux kernel (running this Linux kernel for several weeks already). The mount option 'user_xattr' was already in use for one or two weeks to accommodate 'calendar server' (Apple's open source CalDav server). Also this weekend several packages were updated. I think I did not reboot after installing these updates. So the move of the database to the SSD was the first restart of the SKS-server. The following packages will be upgraded: bind9 bind9-doc bind9-host bind9utils dnsutils libbind9-50 libdns55 libisc52 libisccc50 libisccfg50 liblwres50 libsmbclient libwbclient0 samba samba-common samba-doc smbclient smbfs sudo swat Here the log lines (default debug level 4) of the last working situation and the first problem, after the copy to the new SSD. 2010-06-20 02:26:15 Hashes recovered from ADDR_INET 83.169.43.165:11371 2010-06-20 02:26:15 419EEB36F639BD5CB7FB89D684566FBF 2010-06-20 02:26:15 91FB304214BE8D083426E37480435103 2010-06-20 02:26:15 C165660D7A7D0AA0EFA1198BAC1811DD 2010-06-20 02:26:15 D2F3FCAE96B06E766E891F24997F76C4 2010-06-20 02:26:25 Requesting 4 missing keys from ADDR_INET 83.169.43.165:11371, starting with 419EEB36F639BD5CB7FB89D684566FBF 2010-06-20 02:26:25 4 keys received 2010-06-20 02:26:26 Added 7 hash-updates. Caught up to 1276993585.849555 2010-06-20 02:26:28 DB closed 2010-06-20 05:10:58 Opening log 2010-06-20 05:10:58 sks_recon, SKS version 1.1.0 2010-06-20 05:10:58 Copyright Yaron Minsky 2002-2003 2010-06-20 05:10:58 Licensed under GPL. See COPYING file for details 2010-06-20 05:10:58 Opening PTree database 2010-06-20 05:10:58 Setting up PTree data structure 2010-06-20 05:10:58 PTree setup complete 2010-06-20 05:10:58 Initiating catchup 2010-06-20 05:11:00 DB closed From 2:26 to 5:10, I copied several partitions. So, has anybody a clue why the SKS-server refuses operation? The log lines (at debug level 10 or 100) don't help me further. Are there any known issues with SKS and ext4 or with the last Debian update to bind, samba, etc? Any other things I can try? (Oh, I copied the /srv/pub/sks directories again, now with rsync -xaX instead of cp -a, but with the same result...) Please, help. TIA! Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
Problem solved (Re: [Sks-devel] Slow syncing)
Hi, On 19-02-10 21:28, Arnold wrote: On 08-02-10 20:50, Arnold wrote: On 03-02-10 08:15, John Clizbe wrote: Ryan wrote: I had the same issue develop last year, never figured out why but when the server was unexpectedly rebooted it came back and started working fine. I recompiled/rebuilt the db, everything and nothing worked, it'd get stuck on a loop requesting same keys over and over again never getting anywhere. I retired the server shortly after this reboot and I have not had any issues w/my current setup. The only time I've seen anything like that recently was a problem that looked like a locking issue on the DB files. I setup DB_CONFIG files in both KDB and PTree and haven't had any problem since. [deleted] I figured the key with hash 1ECBAAC1A284D69B93901F676074D62F seems to be missing on my server. To verify, I tried to query that key. Much to my surprise, I found the following URL returns the correct key!!! http://pgpkeys.mallos.nl:11371/pks/lookup?op=hgetsearch=1ECBAAC1A284D69B93901F676074D62F This key has key ID 0x8523592f. Now the following URL *should* also give some result, but is does NOT! http://pgpkeys.mallos.nl:11371/pks/lookup?search=0x8523592Fhash=on So, I guess my +/- 1 missing keys are actually in my database, but not indexed or so. So, I did a key dump of my own database, erased the database and build the database from my own dump. Statistics when I did the dump: Total number of keys: 2791114 After building from the dump: Total number of keys: 2801364 So, my system was continuously trying to receive 10250 keys that were already in the database! After restarting the servers I fetched 677 keys from a peer, once. The log below now shows that the server is up to date :-) Don't know what may have caused it. But I'm happy it's correct now. Arnold 2010-02-21 00:25:09 Beginning recon as server, client: ADDR_INET 195.111.98.30:55383 2010-02-21 00:25:09 Joining reconciliation 2010-02-21 00:25:10 Reconciliation complete 2010-02-21 00:25:10 No hashes recovered from ADDR_INET 195.111.98.30:11371 2010-02-21 00:26:00 Recon partner: ADDR_INET 76.184.64.189:11370 2010-02-21 00:26:00 Initiating reconciliation 2010-02-21 00:26:07 Reconciliation complete 2010-02-21 00:26:07 No hashes recovered from ADDR_INET 76.184.64.189:11371 2010-02-21 00:26:08 Beginning recon as server, client: ADDR_INET 195.111.98.30:36322 2010-02-21 00:26:08 Joining reconciliation 2010-02-21 00:26:08 Reconciliation complete 2010-02-21 00:26:08 No hashes recovered from ADDR_INET 195.111.98.30:11371 2010-02-21 00:27:06 Recon partner: ADDR_INET 83.169.19.225:11370 2010-02-21 00:27:06 Initiating reconciliation 2010-02-21 00:27:06 Reconciliation complete 2010-02-21 00:27:06 No hashes recovered from ADDR_INET 83.169.19.225:11371 2010-02-21 00:28:04 Recon partner: ADDR_INET 76.184.64.189:11370 2010-02-21 00:28:04 Initiating reconciliation 2010-02-21 00:28:05 Reconciliation complete 2010-02-21 00:28:05 No hashes recovered from ADDR_INET 76.184.64.189:11371 2010-02-21 00:29:04 Recon partner: ADDR_INET 76.191.185.172:11370 2010-02-21 00:29:04 Initiating reconciliation 2010-02-21 00:29:24 Reconciliation complete 2010-02-21 00:29:24 No hashes recovered from ADDR_INET 76.191.185.172:11371 signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] Slow syncing
On 03-02-10 08:15, John Clizbe wrote: Ryan wrote: I had the same issue develop last year, never figured out why but when the server was unexpectedly rebooted it came back and started working fine. I recompiled/rebuilt the db, everything and nothing worked, it'd get stuck on a loop requesting same keys over and over again never getting anywhere. I retired the server shortly after this reboot and I have not had any issues w/my current setup. The only time I've seen anything like that recently was a problem that looked like a locking issue on the DB files. I setup DB_CONFIG files in both KDB and PTree and haven't had any problem since. s...@yogi:/var/sks/KDB# cat DB_CONFIG First I tried db4.6_verify on the database files. No luck. Then I removed the PTree directory and rebuilt the PTree - same loop. Now I stopped the server, added the DB_CONFIG file in both the directories (DB and PTree) and still the same looping behaviour, as shown in the log below. Note that it starts at a different place at some(!) times. In the log below, I isolated the line ...starting with 03... In yesterday's log I have 36 lines NOT in the range 1D4... to 1ED... Those lines mostly start with 0... or 1..., but over the last days, 2010-02-05 19:47:02 to be exactly, I also have some lines (one sequence of lines) requesting series starting with 2...to E... Does anyone have any idea? I'm running the binary version packaged in Debian Stable distribution (v1.1.0). I've built the full database (no fastbuild). By looking at the source (also downloaded from the Debian distribution) I get to approx. line 100 in the file 'reconserver.ml', but OCaml is one of the languages I do not speak. What can I do? Is my pseudo random generator out of entropy? Of course I can make a dump of the pgp-keys and rebuild the database from scratch. But, that takes much time and, more important, will it help? Please, advice! Thank you, Arnold 2010-02-08 20:12:12 Requesting 200 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1ECB993CC22AA0A3A57314312D7E0606 2010-02-08 20:12:22 200 keys received 2010-02-08 20:12:31 Requesting 188 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1ED05AC2E9A80D81541C05C8E1D594B7 2010-02-08 20:12:44 188 keys received 2010-02-08 20:12:49 Added 18 hash-updates. Caught up to 1265656365.194739 2010-02-08 20:13:13 Beginning recon as server, client: ADDR_INET 195.111.98.30:44519 2010-02-08 20:13:13 Joining reconciliation 2010-02-08 20:13:15 Reconciliation complete 2010-02-08 20:13:15 10193 hashes recovered from ADDR_INET 195.111.98.30:11371 2010-02-08 20:13:26 Requesting 200 missing keys from ADDR_INET 195.111.98.30:11371, starting with 03B603BB17E6972A935B9AB23CD4F633 2010-02-08 20:13:26 200 keys received 2010-02-08 20:13:27 Added 5 hash-updates. Caught up to 1265656406.936460 2010-02-08 20:13:27 Requesting 200 missing keys from ADDR_INET 195.111.98.30:11371, starting with 1DE93EB2060F37D2227858631F4B0977 2010-02-08 20:13:29 200 keys received 2010-02-08 20:13:29 Requesting 200 missing keys from ADDR_INET 195.111.98.30:11371, starting with 1DEDC2CA5783C6A773960D50668E2E74 2010-02-08 20:13:30 200 keys received signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
[Sks-devel] Slow syncing (was: Re: Memory Leak in recon server?)
On 02-02-10 16:05, Phil Pennock wrote: On 2010-02-02 at 01:19 +0100, Arnold wrote: If it is not in the number of peers, then what can I do further to make it sync more quickly? Do the recon logs contain anything to suggest a cause for not fetching all keys? What about if you bump up the debug level? As far as I can see / understand, it is spending most of the time examining blocks of 100 keys (the hashes I assume) and find they are the same. Current log level is 4 (can set it higher tomorrow if necessary). Below some lines from my log with update activity. Half an hour before it also found a difference... The db.log tells me that I've no e-mail peers, which is indeed the case. One thing I now noted is that most lines of the form Requesting 100 missing keys from ADDR_INET ...:11371, starting with 1ED038D719CE90EEF95B6F185D191EA7 start with 1D. or 1E. Is that OK, or does it mean it is looping somewhere? If I miss or misinterpret something, please let me know! Teun suggested the capacity of my server would be insufficient. Looking at the memory use and CPU load (see below), I guess it will do fine. The average network traffic over the past 9 days is about 2% of my capacity, both upload and download. The statistics at http://www.pool.ntp.org/scores/82.95.191.161 show that the server is accurate in time keeping, synchronising over the net. So I've no network problems either. Jens suggested off-list to add only his server and set http_fetch_size: 500 until my server has caught up. I will try that tomorrow (going to bed now). For now I have added three more servers. However, the log shows some time peer A, then some time peer B, etc. So, more peers may indeed not do the trick, but only spread the traffic across the peers. Well, thank you all! Arnold $ free total used free sharedbuffers cached Mem: 20683282016324 52004 0 1547001445292 -/+ buffers/cache: 4163321651996 Swap: 497972816 497156 $ top top - 01:50:08 up 9 days, 2:31, 3 users, load average: 0.41, 0.36, 0.46 Tasks: 153 total, 2 running, 151 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2068328k total, 2016760k used,51568k free, 154816k buffers Swap: 497972k total, 816k used, 497156k free, 1445292k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 32131 debian-s 20 0 35856 31m 28m S 1.3 1.5 0:53.81 sks 13462 list 20 0 10840 7832 2564 S 0.3 0.4 1:37.98 python 13524 mysql 20 0 126m 24m 5140 S 0.3 1.2 5:09.82 mysqld 1 root 20 0 2100 684 588 S 0.0 0.0 0:08.26 init In recon.log: 2010-02-02 06:54:54 Requesting 100 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1ECDEB960744508BCA9D77259DAFB81D 2010-02-02 06:54:56 100 keys received 2010-02-02 06:54:57 Requesting 100 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1ED038D719CE90EEF95B6F185D191EA7 2010-02-02 06:54:58 100 keys received 2010-02-02 06:55:00 Requesting 83 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1ED274B269B1A93E7434DD098FA12319 2010-02-02 06:55:01 83 keys received 2010-02-02 06:55:02 Added 3 hash-updates. Caught up to 1265090101.513311 2010-02-02 06:55:39 Recon partner: ADDR_INET 76.184.64.189:11370 2010-02-02 06:55:40 Initiating reconciliation 2010-02-02 06:55:43 Reconciliation complete 2010-02-02 06:55:43 10182 hashes recovered from ADDR_INET 76.184.64.189:11371 2010-02-02 06:55:46 Requesting 100 missing keys from ADDR_INET 76.184.64.189:11371, starting with 00C0983B4B12FB65DFE4FB5992282753 2010-02-02 06:55:47 100 keys received 2010-02-02 06:55:49 Added 1 hash-updates. Caught up to 1265090147.860824 2010-02-02 06:55:49 Requesting 100 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1DE771CA7BDC27F9BF991D9DDD3E3A4A 2010-02-02 06:55:50 100 keys received 2010-02-02 06:55:50 Requesting 100 missing keys from ADDR_INET 76.184.64.189:11371, starting with 1DE9463535E7F871D4CCE03BD31B61CE 2010-02-02 06:55:51 100 keys received In db.log: 2010-02-02 06:54:53 Applying 0 changes 2010-02-02 06:54:56 Applying 0 changes 2010-02-02 06:54:57 mail transmit keys error in callback.: Failure(No partners specified) 2010-02-02 06:54:58 Applying 0 changes 2010-02-02 06:55:01 0 potential merges found for keyid 79576865 2010-02-02 06:55:01 1 updates found before filtering 2010-02-02 06:55:01 0 potential merges found for keyid F871868C 2010-02-02 06:55:01 1 updates found before filtering 2010-02-02 06:55:01 0 potential merges found for keyid F30EFBE2 2010-02-02 06:55:01 1 updates found before filtering 2010-02-02 06:55:01 Applying 3 changes 2010-02-02 06:55:01 Adding hash 9BB6A361CD79739D3F2B4259D51CC582 2010-02-02 06:55:01 Adding hash E318F86F4FBF9BED718AFADADCD10E81 2010-02-02 06:55:01 Adding hash FD1CD68BE7B2B5DBAA7A772C562108B8 2010
Re: [Sks-devel] Memory Leak in recon server?
Hi, On 02-02-10 00:15, Phil Pennock wrote: On 2010-02-01 at 16:25 -0500, Daniel Kahn Gillmor wrote: Are you suggesting that if a single peer was to, say, flush its DB and re-connect, it could trigger this memory consumption on all/any of its peers? Would the memory consumption increase proportionally to the number of peers which did this? What I know is that I personally have seen problems in the past with certain peers, discussed on this mailing-list. A couple of peers with an empty DB did not help, but one strange peer could totally kill things. Looking at http://sks.spodhuis.org/sks-peers now, I see that peers of keyserver.cais.rnp.br, pgp.acm.jhu.edu, pgpkeys.mallos.nl, keyserver.ws and sks.ms.mff.cuni.cz might want to cough at those operators and nudge them to take a look at their servers to see if they're healthy. As operator of pgpkeys.mallos.nl, I see it is indeed still 10k keys behind. I downloaded the base set of keys 21 Dec '09 and started peering with keyserver.gingerbear.net 7 days later (thanks John). In my statistics I see a more or less constant rate of 300 to 400 new keys added per day. This is about the same keyserver.gingerbear.net is showing, so my system cannot catch up and will stay 10k behind. I was hoping a second peer would improve this rate, but keyserver.fishysnax.com seems to have died soon after it started on 7 Jan '10. OTOH it is only now that I compare the two rates and conclude that my system will be behind forever. So, yet another call for peers :-) pgpkeys.mallos.nl 11370 # Arnold 0xB66BBBAA This is a private machine, physically located in the Netherlands (EU). The machine has no IPv6 connectivity, yet. For further details visit: http://sks.mallos.nl:11371/pks/lookup?op=stats http://pool.sks-keyservers.net:11371/pks/lookup?op=indexsearch=0x642A49C5B66BBBAA If it is not in the number of peers, then what can I do further to make it sync more quickly? Thank you, Arnold signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
