Re: keyserver.insect.com GDRP takedown request
Thank you for the procedure. For this specific user, he was helpful enough to include the keyids, so it is somewhat easier: - Run the following command to get the keyIds for the blacklist to add:cat < fingerprints.txt | tr A-Z a-z | sed -e "s/^/'/" -e 's/$'"/'/" | tr \\012 ,; echo - Add them to the blacklist first (so they will not be resynced later) - Restart the hockeypuck server to reread the config file - Write the keyids to a file, "fingerprints.txt" - Run the following command to get the rfingerprints rev < fingerprints.txt | tr A-Z a-z | sed -e "s/^/'/" -e 's/$'"/'/" | tr \\012 , | sed 's/.$//'; echo - Run the following to SQL commands with replaced by the output of the above script delete from subkeys where rfingerprint in (); delete from keys where rfingerprint in (); The SQL command for this user (including his keyIDs) will be available for 30 days at https://onice.ch/s/46SJq9ELM9fnHgw . (Not included here, as I do not want to be responsible for his personal information to be archived by the list.) -Marcel Am Freitag, dem 27.05.2022 um 06:48 +0200 schrieb Alexandre Dulaunoy: > Hi All, > > Hockeypuck supports blacklists (from version 2.1.0) when you can list > all the fingerprint keys you want to avoid being synced. > > In addition, you can delete the keys from Hockeypuck (PostgreSQL > database). > > A key can be deleted from the SQL database in the following way: > > - Query the pks interface for the offending key, get the hash > fingerprint from Hockeypuck; > > - Connect to Postgresql via psql > > -select rfingerprint from keys where md5 in (); > > - The returned rfingerprint can be used to delete to delete the > subkeys > delete from subkeys where rfingerprint in (); > - When all subkeys are deleted. > - delete from keys where md5 in () > Don't forget to add the key in blacklist: > [hockeypuck.openpgp] > blacklist=[ > "KEYFINGERPRINT"] > I hope this helps. > > Blacklists -> https://github.com/hockeypuck/hockeypuck/releases > > On Fri, May 27, 2022 at 6:09 AM Allen Zhong wrote: > > Maybe it would be possible for the server to maintain some sort of > > a > > "block list" and reject to receive those keys in the list and also > > not > > returning them to the client? I think that's possible but as it > > requires > > changes of the server software (hockeypuck and sks-server, etc.) > > it's > > not likely to be a quick one. > > > > On 2022/5/27 11:01, Ced wrote: > > > If anyone has an idea to prevent the collapse of the few > > remaining SKS > > > keyservers, please let us know otherwise we'll have to take down > > our > > > server too pretty soon :( > >
Re: State of the graph
Andrew, thanks for the visualization! I'm feeling flattered that keyserver.trifence.ch is at the center of the graph, but this also means it could become a single point of failure (and currently, sks.pyro.eu.org is trying to get back up to speed and has probably requested ~80k keys in past few hours from it). The green network looks way too much like a star topology with keyserver.trifence.ch at the center. It would be great if each of the green (and cyan) nodes could connect to one or two of the other nodes with low degrees. Feel free to use keybath.trifence.ch, but please also try to get connectivity which does not rely on me. -Marcel Am Dienstag, dem 14.12.2021 um 08:39 -0800 schrieb Skip Carter: > It looks like the cluster is collapsing. A year ago I had 18 peers, > now I have > 8. > > On Mon, 2021-12-13 at 10:38 +, Andrew Gallagher wrote: > ... > > It is now apparent that a significant fraction of otherwise > > responsive > > keyservers are not correctly syncing with any of their peers. These > > are > > indicated in the second graph of > > https://spider.pgpkeys.eu/graphs by a > > > > > signature.asc Description: This is a digitally signed message part
Re: hockeypuck runaway
Skip, I noticed a similar error message, but just with one of my peers about a week ago. I did not notice an extremely high load, though (maybe just didn't watch closely enough?). I had looked at some of the read length numbers, then, looking for a potential source of the problem. Here's what I wrote then, in the hope that it might help locating the problem: > * A third option might be that some proxy (or similar) is > intercepting the request and its answer is misinterpreted. However, > the numbers (45083906, 41977124, 57924048) do not look like ASCII > text misinterpreted as binary. However, 1764832326 (the 1.7GB number) > is 0x69313446, which could be "i14F" (or, little endian, "F41i"); > 1577058304, the 1.5GB number, is 0x5E00 (0x5E is "^"), which also > looks like a field mismatch (and not a ptree corruption). > (Given that it is multiple nodes now, the "weird proxy" is probably no longer the cause. However, it is worth noting that some of the lengths do look non-random. Maybe reading from uninitialized memory (or whatever the equivalent is in Go)? -Marcel Am Montag, dem 25.10.2021 um 09:34 -0700 schrieb Skip Carter: > A couple of months ago I had an incident where hockeypuck ran away > resulting in > a CPU load over 50. I firewalled accces to the http port temporarily > and the > load subsided. This has happened a couple times since then again > today. > > I found this in the logs (which I find frustratingly inadequate): > > error[] recon with xxx.xxx.xxx.xxx:36506 failed error=read > length > 804813932 exceeds maximum limit > > The remote address is more than one of my peers. > > Has anyone else had CPU runaway issues ? What is the cause and the > cure ? >
Re: HockeyPuck deployment problems
Michele, it is great to hear about your interest in computer security and I am sorry to hear that you do have problems with Hockeypuck in Docker. I am running two Hockeypucks under Docker and have had some problems in the beginning, but I hoped that what I learned during that process had been reflected in the hockeypuck.io documentation. Can you tell us more about your problems, i.e., what is your current setup and what does and does not work and what you already tried to diagnose these problems or even fix them? -Marcel PS: How do you plan to check for the split-world behavior? (And I hope you will not use this against the global keyserver network… BTW: Contributions for how to avoid this are welcome.) PPS: At some point, it may be useful to take the discussion off-list. But right now, the information might still be relevant to the original goal of the list, contribute to the development and operation of (SKS[- protcol] based) OpenPGP keyservers. Am Montag, dem 25.10.2021 um 10:00 + schrieb Michele Marazzi: > Hi,I’m a student of the Politecnico di Milano university and I’m > proceeding in my road to the master degree in computer science and > engineering. > I have inserted in my academic plan a course called “Cryptography and > architectures for computer security” in which the teacher proposed > some projects concerning the argument of the course. > In particular one project is about HockeyPuck and this captured the > interests of me and a colleague of mine. > This is the draft of the project: > “HockeyPuck (https://hockeypuck.io/) is a recent reimplementation of > the traditional SKS keyserver software, employed to build a > distributed storage for openPGP certificates. The synchronization > mechanism between keyservers may lead to split-world scenarios if a > malicious upload is performed. > The purpose of this project is to test the resilience of HockeyPuck > against this attack.” > We have tried many times to install and configure HockeyPuck in our > machines and cloud machines, but never succeed. > As recommended by some contributors, we used Docker for the > installation process, and the scripts associated to that also deal > with the automatic configuration and installation of Postgresql, > Nginx and Prometheus. So we did no more than running all the scripts > correctly and then using docker compose (after having downloaded the > keys). > We also tried with the snap in the snap store, but never succeed also > with that. > We used mainly Virtualbox with Ubuntu installed (we used at first the > last version, then we tried also with 20.04), but never succeed. > We have followed the documentation on Github and on the old > HockeyPuck page made by Casey Marshall (no more maintained), but the > results never changed. > We wrote also issues on the Github repo, but the answers were not > useful to complete the deployment. > Our teacher have recommended to wrote you an email, since you are the > community referencing to HockeyPuck, to ask some help for the > configuration process, in particular we are searching for some > detailed guidelines (and also updated sources, if needed) both for > the installation and the usage. > I cordially thank you in advance for the help. > Best Regards. > > Michele Marazzi
Re: High rate of updated keys
I have not investigated closely, but I noticed that after a restart of the Hockeypuck server, several hundred "updates" are being processed (I am using the version which does negative caching of recon attempts which did not result in updates). So, maybe we need to look closer at what actually counts as an "update" for Hockeypuck. An alternate explanation would be that the changes being applied are not idempotent or cumulative, e.g., that some UIDs or signatures are being reordered, which is counten as an update. And, in theory, several of those changes might be circulating in the network, updating each other again and again (even though I guess that over time, some of the updates should "win" over the others). But as I said, I'm just guessing wildly right now. -Marcel Am Dienstag, dem 11.05.2021 um 15:10 +0100 schrieb Andrew Gallagher: > On 08/05/2021 20:23, Andreas Puls wrote: > > Im running on both of my servers the deafult value. > > I think the main difference between pgpkeys.eu and other hockeypuck > installs is that pgpkeys.eu recons directly with three of the biggest > SKS pool members, while most other hockeypuck installs do not. I > temporarily disabled recon with all SKS peers and the number of > modified > keys immediately fell back to normal levels. > > I suspect this may be related to the hockeypuck/SKS recon thrashing > problem that Marcel diagnosed - the number of updated keys does not > seem > to reflect actual updates but rather update attempts, and as the SKS > and > Hockeypuck datasets have diverged, so have the number of repeated > sync > failures. This would appear to have been growing slowly for some > time. > > I'm still not sure why pgpkeys.eu has been disproportionately > affected - > other hockeypuck nodes have several SKS peers and haven't suffered to > the same extent. I suspect it may be an artifact of the topology - > perhaps pgpkeys.eu is getting different update sets from two > different > sources and they keep overwriting each other, or some such. > > Investigations continue... :-) > signature.asc Description: This is a digitally signed message part
Re: Key diff anomaly
Gabor, all, the numbers are positive again, but the anomaly still persists (UTwente 73k ahead of the active keyservers). It seems that from the active set, only Andreas Puls and escomposlinux are being peered with. I hope the operators of these three nodes can have a look at their peering and the key difference. -Marcel PS: Additionally Bcc-ing those three, in case their sks-devel mail is pre-sorted into some obscure folder. On Mon, 2021-04-05 at 07:46 +0200, Kiss Gabor (Bitman) wrote: > I've just noticed that key diff of ALL current 26 pool members is > negative. > Meanwhile keyserver.snt.utwente.nl is dropped from the pool > however it seems to be absolute healthy. Except its key diff: 73432. > > I guess this this node got an attack-like burst of keys from outside. > (And this 73k extra keys distorted the average so much that everybody > else > seems to be lacking thousands of keys.) > > Gabor > signature.asc Description: This is a digitally signed message part
Re: Pool dried up
> > > > > > Looking at the cached metadata it appears that when the spider > > > > ran, > > > > pod02.fleetstreetops nodes was unavailable, as was > > > > pgpkeys.co.uk > > > Apologies, I didn't mean to cast doubt on the reliability of your > > node, > > but rather on that of the spider. It does not maintain much of a > > historical record, and so depends on a single measurement per node > > each > > > yes, that's the case with sks.infcs.de as well. > > That server sometimes handles a reconcil task that it does not return > the stat-page in time. Per log, the server is running the whole time, > but the spider cannot retrieve the stat page all the time, because of > the sequential design of SKS server. > Would you "core server people" consider caching the status page in the proxy? (I do fetch the status page every 5 minutes from the backend servers to create a "homegeneous" view of the pool, as each is configured slightly differently; and serve that statically from the load balancer.) -Marcel signature.asc Description: This is a digitally signed message part
Re: Lying about Hockeypuck being SKS?
pgpkeys.eu is in the pool, however, the "Software" table cell is red. Further investigation showed that they spell "sks" in lower case, unlike the real SKS. Verifying against the source code [1], the server selection code is a blacklist: "SKS" < 1.1.6, "GnuKS", and "hockeypuck" are blacklisted (case-sensitive). How about instead of claiming to be "SKS", Hockeypuck servers just claim to be "Hockeypuck" (title case instead of lower case)? That would be kind of a "white(list) lie". They will have a red cell, but that will not preclude their I have changed my Hockeypucks to return this version and will check in 30 minutes whether the theory can be corroborated. Happy keyserving, -Marcel [1] https://git.sumptuouscapital.com/?p=sks-keyservers-pool.git;a=blob;f=sks-keyservers.net/status-srv/sks-status.inc.php;h=972bb5b56412ae54b8aade234ea02bb8c9545d45;hb=HEAD#l309 On Mon, 2021-03-22 at 21:13 +0100, Andreas Puls wrote: > > > Am 22.03.2021 um 20:41 schrieb Marcel Waldvogel: > > On Sun, 2021-03-21 at 22:56 +0100, Andreas Puls wrote: > > > > > > I've created now a patch that just replaces in the json export > > > contact > > > with server_contact and Total with numkeys. > > > https://github.com/apuls/hockeypuck/commit/34fbdfcf73b60e6001f3770b86d8750d1c8b5385 > > > > Great, thanks! I just merged this. Now my Hockeypuck server appears > > in > > the statistics. > > > You're welcome! > > > > In my hockeypuck configuration i've set Version to 1.1.6+ and > > > Software > > > to SKS > > > Yeah, i've done it too. :) > > > Hockeypuck is blacklisted in the sks-keyservers.net code, because > > it > > was not good enough to be incorporated into the pool when Kristian > > wrote the code. Now, it seems to be in the same ballpark as SKS. > > > > Asking Kristian to remove the Hockeypuck ban resulted in him > > explaining > > that he does not plan to change the code or accept changes; > > instead, we > > should set up our own fork of his code. > > > > I think this leaves us with the following ways to progress: > > > > a) We leave it as is, Hockeypuck is fine, but just not in the pool. > > b) We create a second pool, where Hockeypuck is acceptable (and > > probably SKS as well). > > c) We agree that Hockeypuck lying to be SKS is accepted in the > > pool, > > and maybe even recommended. > > > > I would favor (c), plus keeping the version number in the 2.x > > range, so > > that experts still can tell the difference. > > > b would be great but i think this is a hell of work. > > Since we haven't heard for a while from Kristian and the pool is > working > - ok more or less - i would go with option c too. Also with the > Version > string 2.x . > > Opinions? > We need to fix the peers field which will be reported via options=mr > to > meet the requirements from the pool skript. > > > -Marcel > Br > Andreas >
Re: An evil idea :-)
Gabor,
so, please call me Mr. Evil ;-)
A few weeks ago, I set up a simple Nginx load balancer (two lines with
https-portal[1]) statically seeded with the nodes that were in the pool
at that time for test purposes. It randomly returns the status page of
one of the backend servers, though, but that could be easily changed.
I wasn't as evil-minded to start faking a pool that would gradually
fake an increasing delta of keys against the "real" keys. Kudos for
that! ;-)
(The idea was triggered by the general unreliability of pool members. I
think we need to work on that. And also spam, trust, GDPR compliance,
and RTBF; but these are topics for a different thread.)
Greetings,
-Marcel
PS: Even if you are just load-balancing your own servers, you might
include the following line into your Nginx load balancer config
("non_idempotent" is fine, as even the POST requests that modify
anything, notably /pks/add, are in fact idempotent):
proxy_next_upstream error timeout http_500 http_502 http_503 http_504
http_404 http_429 non_idempotent;
[1] https://github.com/SteveLTN/https-portal
On Mon, 2021-03-22 at 21:08 +0100, Kiss Gabor (Bitman) wrote:
> One can decide to setup a proxy server without any own backend
> but redirecting queries to some of the existing servers.
> No one would recognize the cheating. :-)
>
> Gabor
> --
> "Virgil Brigman back on the air" (Abyss)
>
Lying about Hockeypuck being SKS?
On Sun, 2021-03-21 at 22:56 +0100, Andreas Puls wrote: > > I've created now a patch that just replaces in the json export > contact > with server_contact and Total with numkeys. > https://github.com/apuls/hockeypuck/commit/34fbdfcf73b60e6001f3770b86d8750d1c8b5385 Great, thanks! I just merged this. Now my Hockeypuck server appears in the statistics. > In my hockeypuck configuration i've set Version to 1.1.6+ and > Software > to SKS Hockeypuck is blacklisted in the sks-keyservers.net code, because it was not good enough to be incorporated into the pool when Kristian wrote the code. Now, it seems to be in the same ballpark as SKS. Asking Kristian to remove the Hockeypuck ban resulted in him explaining that he does not plan to change the code or accept changes; instead, we should set up our own fork of his code. I think this leaves us with the following ways to progress: a) We leave it as is, Hockeypuck is fine, but just not in the pool. b) We create a second pool, where Hockeypuck is acceptable (and probably SKS as well). c) We agree that Hockeypuck lying to be SKS is accepted in the pool, and maybe even recommended. I would favor (c), plus keeping the version number in the 2.x range, so that experts still can tell the difference. Opinions? -Marcel signature.asc Description: This is a digitally signed message part
Seeking peers for keyserver.trifence.ch
Hi, I started running a keyserver (again) after some 20 years of break (then still running the first generation of public keyservers) and am looking for peers, to remain in sync. I provide two different endpoints for SKS and Hockeypuck (see below for explanation): SKS servers are welcome to peer with: keywin.trifence.ch 11370 # Marcel Waldvogel 0x9CF85070DD5B7293B6988379C3C53A69327FB3DC while Hockeypuck servers should peer with: # Marcel Waldvogel # 0x9CF85070DD5B7293B6988379C3C53A69327FB3DC [hockeypuck.conflux.recon.partner.hkp-winterthur] httpAddr="keyserver.trifence.ch:11371" reconAddr="keyserver.trifence.ch:11370" Why two servers? I had set up Hockeypuck, but then learnt the hard way that recent Hockeypuck with their anti-DoS measures can create a lot of traffic between Hockeypuck and their SKS peers. While Hockeypuck now essentially no longer requests keys which will be shrinked by anti-DoS, their SKS peers will still regularly request those abusively huge keys. So the two servers above handle all the overhead inside the same physical machine and are configured to reduce the load they generate on each other. -Marcel PS: More information: https://github.com/hockeypuck/hockeypuck/pull/107 and https://github.com/hockeypuck/hockeypuck/issues/108 signature.asc Description: This is a digitally signed message part
