Well, I guess I'll chime in. We had a RaQ4 (raid) that when bad last week. I don't know if it was a hack or hardware failure. Around 2 pm Tuesday, the machine went dead to we requests. Luckily I had a ssh tunnel open (for using IMAP) and started trying to figure out what when wrong. But as I started typing commands I was getting alot of "command not found" from the command prompt. Well the entire /bin directtory was gone!!, /etc was empty. and /var was empty also.
Well I wasn't feeling so happy. After the ISP tried to fix it with some disk tools and gave up. I went over and got the hard drive. Took it home, mounted it up to another linux box and got all of /home off (thankyou Jesus!!). I had an 10 month old backup in the form of a raq migration utility dump. I returned the harddrive and they reinstalled the machine. And I patched it with all of the latest. After uploading the mu file and running it (1.2 gig). and then uploading a tar.gz of the /home (2.2 gig) I had the machine limping along pretty good. But then it started to die again. I was on console and I think it was the 'whereis' command I ran got a "command not found". A quick cd to /dir showed that it only had half the files it should have. I immediatily shutdown the machine to prevent further data loss. This was now Wendnesday night. Two failuers or possible hacks in two days!?!?!? ugh. I don't know if I got hacked again, or it is a hardware failure. Well to keep the story shortish, we decided to abonded the RaQ completely, got set up on a P4 Hyper thread box with RH9. Got all 120+ domains set up on it. My isp was able to reroute all 100 ips to the new box. Instead of using sendmail, I set up postfix. Instead of proftpd, we are on vsftpd now. I've kept the same directory structure for the RaQ and I am considering writing a control pannel to handle web/users/ftp/email for the new box. But none of our customers really ever used it anyway. Over all I'm sad that I had so work so much to get the server back up, but I'm happy to have the cobalt behind me. Goodbye cobalt, Lance Plug for my isp: Here is the link for the new server http://www.cari.net/Apps-template/apps-template.html?service_key=141 we went too. ----- Original Message ----- From: "Jelmer Jellema" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, November 06, 2003 01:54 Subject: RE: [cobalt-security] Re: Cobalt Raq 4 Hacked > Tripwire works just fine, though it will take you a while to get the right > policy-entries in (if I recall correctly starting with the redhat version of > a prefab policy file saves you some time but if you use the > cobalt-management admin system you should add stuff). > > I run it from cron every x hours and it reports nicely everytime I changed > something. Ofcourse I cannot know whether it failed to report some hack, but > I still feel secure (as did you until...). > > I could send you - off list - a policy file I use on a Raq4, with some > site-specifik stuff out. But as I don't use the cobalt management stuff any > more, this is not very well included. > > I take it you also have chkrootkit installed. > > Jelmer > > ----------------------------------------------------------------- > Jelmer Jellema - Spin in het Web > http://www.spininhetweb.nl > Spin in het Web: Alle Touwtjes In Handen > ----------------------------------------------------------------- > > > > -----Oorspronkelijk bericht----- > > Van: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] Namens Cory > > Hollingsworth > > Verzonden: woensdag 5 november 2003 16:46 > > Aan: [EMAIL PROTECTED] > > Onderwerp: [cobalt-security] Re: Cobalt Raq 4 Hacked > > > > > > > Subject: [cobalt-security] Cobalt Raq 4 Hacked > > > > > > Hi all. > > > Today at 2:00 in the morning our RaQ4 was hacked. > > > > > > The hack replaced all files named index.* with own > > > hacker content. > > > > > > We have had this issue a couple of weeks ago. That > > > time the hack afected all index.* files under > > > /home/sites. > > > > > > Now the hack affected all index.* files under / so the > > > Control Panel is affected too. > > > > > > I must note a week ago the server wass rebuilded formm > > > scratch and ALL upgrades were applied. > > > > > > I have chkrootkit and portsentry installed. None of > > > those reported strange activity. > > > > > > I have checked open ports in the server and theres' no > > > strange ports opened. > > > > > > I want to know if somebody has experienced the same > > > issue, and any help will be apreciated.. > > > > > > TIA > > > Pablo > > > > I've had a similar problem with a Raq 4 last week. Instead > > my issue was someone using the server as a DOS attack > > machine. Their attack removed /var/log and turned off all > > network services into the machine. Also unbenounced to me on > > an earlier date Apache stopped logging in the > > /home/sites/XXX/log sub directories. > > > > Even though the machine was fully patched, the Apache logging > > had halted before I applied the latest patches. It may have > > been compromised months before it was used as a DOS attack machine. > > > > Their DOS attack I believe unintentionally wiped out the root > > filesystem so the machine could no longer boot. It took a > > couple of days of hacking at the serial console to get the > > machine working well enough to ssh the accounts and data off > > the machine. > > > > I've since moved everything onto a Raq550 I had hanging > > around as a spare. The only things lost were the users > > passwords as I couldn't easily load them from the old shadow > > file as the Raq550 stores passwords in a BTREE in /var/db/shadow.db. > > > > As of yet I haven't figured out how the machine was > > exploited. But we had 600+ users on the machine with CGI, so > > it is not infeasible the exploit could have been in user CGI > > rather than the core services. > > > > Any one have any luck with tripwire on Raqs? That is the > > only thing I can think of trying at this time. > > > > _______________________________________________ > > cobalt-security mailing list > > [EMAIL PROTECTED] > > http://list.cobalt.com/mailman/listinfo/cobalt-security > > > > _______________________________________________ > cobalt-security mailing list > [EMAIL PROTECTED] > http://list.cobalt.com/mailman/listinfo/cobalt-security > _______________________________________________ cobalt-security mailing list [EMAIL PROTECTED] http://list.cobalt.com/mailman/listinfo/cobalt-security
