-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brian Phillips wrote: > You are right. It actually happened earlier this week, it was Monday or > Tuesday if I remember. I know because I was the one who took the call and > notified the network engineers. I had to go to class, but the resolution > was so: > > The CS department found the port 80 of their inet proxy was being blocked by > OIT's connection. They figured this out because by bypassing their proxy, > they were able to use web traffic. The network engineers started looking at > changes on our end, but they hadn't made any. The CS department called in > later (I think it was an hour later...) and said they had changed the ip of > their proxy to an address that was 3 from their last address. They had port > 80 connectivity for about 5 minutes and it was gone. The network engineers > checked (here's where it gets over my head) and found that something behind > the CS department proxy was sending out network traffic strange enough over > port 80 that the automated system blocked it. Either it was too much, or it > was a malware/virus related traffic, but the system auto-blocked it. The > sysadmins from the CS department said they would investigate more on their > end and turn off whatever was going on. Time to resolution was less than 3 > hours.
I would consider this an accurate view of what happened as seen from the ops center, and as entered into the help ticket system. Unfortunately, it was not the end of the story. The problem kept recurring due to BYU's IPS detecting a "high number of connections" from one "host" over port 80, and automatically blocking that "host". The "host" was the department's transparent proxy which was aggregating the traffic of a thousand client computers, and the "high number of connections" was just the sort of traffic you'd expect with a thousand computers or when 200 students get out of class and all hit slashdot, cnn, espn, and other common sites all at once. The actual resolution involved several attempts to modify the threshold in the IPS, followed by a whitelisting. Any other departments experiencing such problems should pursue a similar course. > But since OIT just turns port 80 off for an entire department at random, it > had to be their fault... They also work entirely too slow... Well, in this particular case, it was OIT's fault, and it was too slow, though my experience has frequently shown this not to be true. The IPS was set to trigger on perfectly valid and expected traffic loads. By blaming OIT, however, I would like to point out that our experience with the vast majority of OIT employees in the resolution of this problem was extremely positive. I place the majority of the blame for both the problem and the delay in fixing it on one particular individual. In all, we experienced web downtime for more than a day on this particular issue. It doesn't show that way in the ticket because apparently the engineer responsible doesn't bother to use the help ticket system. > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of TuxGirl > Sent: Saturday, December 03, 2005 11:56 PM > To: BYU Unix Users Group > Subject: Re: [uug] BYU net authentication. > > >>Interesting... How long ago were you blocked for having multiple SSH >>connections? Were you notified? How long did it take them to block you? >>I've never heard of that policy, but I don't work in the securities group >>so... This was another situation with the IPS, and where the security group (is there really a group?) was at fault. The problem was trivial to reproduce, and the "offending host" could trigger the block instantly. No, there was no notification involved. The resolution of this problem also took took several days. As with most of our other issues, we had an extremely positive experience with the vast majority of the engineers we worked with. With this problem, the IPS threshold was set at 6 ssh connections initiated within 2 seconds, and was repeatedly being triggered due to the fact that all the computers in each of our computer labs aggregates traffic through a single IP. Who woulda thunk that 40 linux boxes behind a single IP could ever have initiated more than 6 ssh connections within a short period of time? We also saw the block triggered by a single host through a number of other perfectly legitimate methods (such as opening several files in konqueror via fish://). At our insistence, the threshold is now much higher, but is still in effect. We are still waiting to see whether we see problems with the current setting. So far so good. Frank - -- Frank Sorenson - KD7TZK Systems Manager, Computer Science Department Brigham Young University [EMAIL PROTECTED] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFDkrrdaI0dwg4A47wRAppRAJ9Am0eH0/Aco/tgBWpOKdp1hKSzSQCg8FUZ OMiTXYPsxsDnDpzlMyRPk7o= =xTIx -----END PGP SIGNATURE----- -------------------- BYU Unix Users Group http://uug.byu.edu/ The opinions expressed in this message are the responsibility of their author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list
