The problem was not DDoS, I think. Our speculation is that it might have been a software/hardware issue possibly (kernel panic, a total OOM and grinding to a halt is also possible I suppose.)
Hetzner replaced the RAM and upgraded the BIOS on the machine as it was not responding to any keystrokes in DC 19. It is now running smoothly again, but who knows when it might strike. In light of this, using Rackspace resources (they graciously donated free cloud services, up to $2000 USD a month at the moment,) I have rebuilt a new backup machine running FreeBSD 9.2 using ZFS. It has 4TB of space, 16GB of RAM and generally should be enough. This is a serious just-in-case, and I will be rewriting the backup scripts to use this machine quite soon, and give the other admins access (sadly I think backups may have silently broken at one point, but I'm not 100% sure.) I'll also be looking to use Rackspace to host an official hackage mirror. To do this safely we'd technically need package signing in place, but it could also be a blessed instance (under the haskell.org domain w/ SSL enabled) for the moment, outside of Hetzner and in a different availability zone (say, the US or Australia.) This should also mitigate the impact for a large variety of users in the wild. I've CC'd Duncan so he's aware (although I'm sure he's on this list.) We'll also still have enough capacity to run GHC buildbots too. Finally, I'm also willing to continue with Hetzner and they have worked wonderfully for me personally, and also us in the past (lambda has an uptime of nearly a year!) However, I think splitting up some of the services is sensible where possible for critical stuff, so I'm taking the first step to that (off-site backups and a mirror.) Others can come as needed. On Sun, Nov 17, 2013 at 6:46 AM, Yitzchak Gale <[email protected]> wrote: > I agree with Mark. Suite Solutions is are using Hetzner extensively > and we are very happy with them. It's a tremendous value. > > Thanks to Gershom for the link to the Y thread though. I'll make > sure that our IT dept. knows to have a contingency plan in > case we are specifically targeted. We should do the same here > in the Haskell community as Jason suggests, unless we feel we > can afford a service that provides what Hetzner provides plus > DDoS protection. > > > On Sun, Nov 17, 2013 at 6:35 AM, Mark Lentczner > <[email protected]> wrote: >> So, I have a hetzner machine, as do others I know in the community - and the >> service has been stellar. >> >> The ycombinator thread makes it clear: The problem is DDoS, and of course >> Hetzner provides no protection against that - which is expected. Is there >> any reason to believe that we will be protected from DDoS at osu? >> >> Do we have reason to believe that this is a Hetzner problem? Do we know the >> root cause of our going down? >> >> - Mark >> >> _______________________________________________ >> haskell-infrastructure mailing list >> [email protected] >> http://community.galois.com/mailman/listinfo/haskell-infrastructure >> > _______________________________________________ > haskell-infrastructure mailing list > [email protected] > http://community.galois.com/mailman/listinfo/haskell-infrastructure -- Regards, Austin - PGP: 4096R/0x91384671 _______________________________________________ haskell-infrastructure mailing list [email protected] http://community.galois.com/mailman/listinfo/haskell-infrastructure
