on May 22 Our server was broken into by some one in China. How it happened is that we had had a hole in our firewall so employees could access out server from the field. This had worked pretty well - until the A&T Motorola modem died and they install two new ones and left the port to the ssh open. Every day at noon we saw a high usage of our Internet for an hour or so. Depending on what they were downloading. We found the bogus IPTABLES in / and /boot that were linked to /etc/? We rebuilt the system from scratch several times and within hours it was back. The last time we were careful to examine the partitions in the drive that has the OS on it. The sizes did not add up and we could not see the hidden partition on the OS drive at the end of the normal partitions. We found the data file in /tmp/.tmp and while looking at its contents we recognized a number of files. The server was unplugged from the Internet and we changed the "root" password and then examined the /tmp/.tmp file and the beginning line was PASSWD: "new encrypted passwd" ready to sent.. From or inability to find the where the SW was lurking we did a low level format of the OS disk, it took a long time to to format a 1/2 T drive. In the mean time we closed all firewall ports in the Modem. Installed the OS again and everything has been working well.
We used a trace route program and in 4 hops we were some where west of Singapore. The really bad problem is that they had downloaded all of our banking information and had attempted to send money from Company savings account to somewhere. Credit Union could not or would not say as to where the money was to be sent. Credit Union has one neat restriction is that you cannot send money from a saving account but must move into you checking. We had to change all account numbers, pass words, id checks etc. We had to buy new checks, stamps for 5 accounts. Several people had their personal banking data on server too. A big pain in the posterior. We are still working on how give guys access to some accounts. The people who did this job had more than a working knowledge of networks, Linux and files systems. We were wondering how they could create a directory at end of file system was a puzzle. They had root privilege, ssh, and with access to bash they were in. How did they covered their tracks so well? "messages" was there but filled with nonsense and file in /var/log that tells you who and what was sent was touched was now missing. "security" was there and you could see the repeated access attempts to break in again. "cron" was changed so daily backups were done after they down loaded all new files. "crontab -e" no longer worked. We made a copy of the OS onto old disk and removed disk from the system. There were so many charges to the OS and files in /etc that we did not even try to repair it. There were 1000's of differences between new install and copy of old system. I personally think the bash problem is over blown because they have to get threw modem, firewall, ssh before they can use "bash". One question remains and that is what code and script did they use to run the system?? If anyone wants details and IP's I will send it to them on an individual basis. We contacted the FBI and after a telephone interview, they were sort of interested but I think the problem is so big they don't have time to work little stuff. This is a little disjointed because it happened over a long time. Larry Linder