I would agree. I didn't have the ticket system in front of me, but everything said here jogs my memory enough and I would say Frank has finished the story where I got fuzzy.
>At our insistence, the threshold is now much higher, but is still in >effect. We are still waiting to see whether we see problems with the >current setting. So far so good. I can see why your department would need exceptions, as well as the Chem department. Anytime there is a firewall, and especially a transparent one, it should be given a special consideration. I can also see why the network engineers would have been cautious about whitelisting your proxy. If I were a network engineer I would be wary about whitelisting a proxy or a firewall just because if you whitelist that, you have effectively whitelisted a couple hundred hosts. I would think that when you requested your whitelist you had to assure OIT that you would be in charge of security on the other side of that firewall before they would approve of the change, a change I wouldn't think unreasonable if given your assurances. I think it would be a safe assumption for us all to say it takes only 1 person to cause us an unpleasant experience, even though our dealings with the department as a whole have been good. >From my point of view, I've had to deal with a few CSR's around campus who felt they knew more about the network than I did. They would also get lost when I tried to explain the concept of a vlan, something I've known since day 1 of my training. But that doesn't mean I should go around blaming the entire department that CSR works for, or all CSRs for that matter. In fact, most I've talked to have been a great help (except for the one who didn't know what an 'ipconfig' was). I could easily say that all on-campus housing residents are impatient and snobby, but then again, out of the 6 I talked to for a resolution two weeks ago, I apologized for it taking us so long to get to the resolution and 5 of them said "No way, you called us back in 10 minutes!" (Even though for some, it was actually like, 20-30.) The last one was the only one who felt so inconvenienced from his 10 minutes of downtime (because it was fixed after 10 minutes and he was one of the first notified) that wasn't even caused by OIT that he felt he had to register a complaint. I guess since OIT is as large a department as it is, there's more than enough chances to drop the ball, and hence the bad rep. Especially around a crowd that actually knows more than the first level of OIT. But the first level wasn't put in to help the elite, it was put in to help the other 90% of the student body. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Frank Sorenson Sent: Sunday, December 04, 2005 2:46 AM To: BYU Unix Users Group Subject: Re: [uug] BYU net authentication. -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brian Phillips wrote: > You are right. It actually happened earlier this week, it was Monday or > Tuesday if I remember. I know because I was the one who took the call and > notified the network engineers. I had to go to class, but the resolution > was so: > > The CS department found the port 80 of their inet proxy was being blocked by > OIT's connection. They figured this out because by bypassing their proxy, > they were able to use web traffic. The network engineers started looking at > changes on our end, but they hadn't made any. The CS department called in > later (I think it was an hour later...) and said they had changed the ip of > their proxy to an address that was 3 from their last address. They had port > 80 connectivity for about 5 minutes and it was gone. The network engineers > checked (here's where it gets over my head) and found that something behind > the CS department proxy was sending out network traffic strange enough over > port 80 that the automated system blocked it. Either it was too much, or it > was a malware/virus related traffic, but the system auto-blocked it. The > sysadmins from the CS department said they would investigate more on their > end and turn off whatever was going on. Time to resolution was less than 3 > hours. I would consider this an accurate view of what happened as seen from the ops center, and as entered into the help ticket system. Unfortunately, it was not the end of the story. The problem kept recurring due to BYU's IPS detecting a "high number of connections" from one "host" over port 80, and automatically blocking that "host". The "host" was the department's transparent proxy which was aggregating the traffic of a thousand client computers, and the "high number of connections" was just the sort of traffic you'd expect with a thousand computers or when 200 students get out of class and all hit slashdot, cnn, espn, and other common sites all at once. The actual resolution involved several attempts to modify the threshold in the IPS, followed by a whitelisting. Any other departments experiencing such problems should pursue a similar course. > But since OIT just turns port 80 off for an entire department at random, it > had to be their fault... They also work entirely too slow... Well, in this particular case, it was OIT's fault, and it was too slow, though my experience has frequently shown this not to be true. The IPS was set to trigger on perfectly valid and expected traffic loads. By blaming OIT, however, I would like to point out that our experience with the vast majority of OIT employees in the resolution of this problem was extremely positive. I place the majority of the blame for both the problem and the delay in fixing it on one particular individual. In all, we experienced web downtime for more than a day on this particular issue. It doesn't show that way in the ticket because apparently the engineer responsible doesn't bother to use the help ticket system. > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of TuxGirl > Sent: Saturday, December 03, 2005 11:56 PM > To: BYU Unix Users Group > Subject: Re: [uug] BYU net authentication. > > >>Interesting... How long ago were you blocked for having multiple SSH >>connections? Were you notified? How long did it take them to block you? >>I've never heard of that policy, but I don't work in the securities group >>so... This was another situation with the IPS, and where the security group (is there really a group?) was at fault. The problem was trivial to reproduce, and the "offending host" could trigger the block instantly. No, there was no notification involved. The resolution of this problem also took took several days. As with most of our other issues, we had an extremely positive experience with the vast majority of the engineers we worked with. With this problem, the IPS threshold was set at 6 ssh connections initiated within 2 seconds, and was repeatedly being triggered due to the fact that all the computers in each of our computer labs aggregates traffic through a single IP. Who woulda thunk that 40 linux boxes behind a single IP could ever have initiated more than 6 ssh connections within a short period of time? We also saw the block triggered by a single host through a number of other perfectly legitimate methods (such as opening several files in konqueror via fish://). At our insistence, the threshold is now much higher, but is still in effect. We are still waiting to see whether we see problems with the current setting. So far so good. Frank - -- Frank Sorenson - KD7TZK Systems Manager, Computer Science Department Brigham Young University [EMAIL PROTECTED] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFDkrrdaI0dwg4A47wRAppRAJ9Am0eH0/Aco/tgBWpOKdp1hKSzSQCg8FUZ OMiTXYPsxsDnDpzlMyRPk7o= =xTIx -----END PGP SIGNATURE----- -------------------- BYU Unix Users Group http://uug.byu.edu/ The opinions expressed in this message are the responsibility of their author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list -------------------- BYU Unix Users Group http://uug.byu.edu/ The opinions expressed in this message are the responsibility of their author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list
