On Sun, 2005-12-04 at 02:46 -0700, Frank Sorenson wrote: > The problem kept recurring due to BYU's IPS detecting a "high number of > connections" from one "host" over port 80, and automatically blocking > that "host". The "host" was the department's transparent proxy which > was aggregating the traffic of a thousand client computers, and the > "high number of connections" was just the sort of traffic you'd expect > with a thousand computers or when 200 students get out of class and all > hit slashdot, cnn, espn, and other common sites all at once. > > The actual resolution involved several attempts to modify the threshold > in the IPS, followed by a whitelisting. Any other departments > experiencing such problems should pursue a similar course.
In Chemistry we have also had problems with this from time to time. Too much traffic from one host. Each time this happened we would gently remind them that our entire department (which is bigger than most colleges) accesses the internet through our firewall, which is just one host. At one time it became such a problem (weekly calls), that we went into QIP and set the name of our firewall to "firewall.byu.edu." That seemed to work. During the last year we upgraded our firewall and now use a pool of out-going ip addresses, so we haven't had any problems with "unusual" traffic recently, except for an FTP issue was turned out to be some weird chinese peer-to-peer program that we found and turned off. We have had many many problems with OIT (people problems mainly, not necessarily network problems) and they come down to a few things. One is lack of communication within OIT. Our agreements with Kelly McDonald often don't seem to filter down to the middle managers. They seem to question our running our network at every step of the way. Because of that we have to frequently go over their heads and talk to Kelly to get things done, which angers the middle managers even more. OIT also sees itself as a profit-driven utility provider but without any competition, rather than a service organization. There is also a perception by those of us outside OIT that there are folks inside OIT who are tying to protect their little empires, whether it be wireless, surplus, two-way radios, or even cell phones. The cell phone incident came up recently as someone was explaining that their provider (who sells those walkie- talkie phones--not sure which one that was) has pretty poor coverage on campus. They wanted to come in and set up some towers which would simply require power and an out-bound network connection to send the voip traffic to their network. According to the cell rep, OIT flatly refused to allow this. However not long after that, Cingular came on campus and put up stations that are using OIT network lines. This person's impression was that because OIT likes to rent out two-way radios at a pretty high price, allowing this other cell phone provider on campus would cut into their market by allowing departments to use cheap, two-way phones. Now this may not be true, but these types of things are the perception of almost everyone I talk to on campus regarding OIT and OIT services. And, as they say, perception is everything. That said, this is in no way an attack on you Brian. We appreciate that you are doing your best in your job to help serve the mission of BYU. In fact many of us (including me) who have really bad-mouthed OIT at times actually worked for Kelly Flanagan. And we have high hopes for OIT now that he is at the helm. As for details on the CS department's problem with ssh, I imagine it is fixed now, since Klark and Frank finally were given a way to reliable duplicate the problem (which plagued them for the last couple of semesters) and I think were able to take the problem to ops. I'm not sure OIT must always be cursed if they do, cursed if they don't. It is possible to keep people happy. If our experience in Chemistry is any indication, decentralization is the key, whether that be distributed to a college or departmental level, depending on the needs. In fact almost all campuses I see around the US are decentralizing their IT infrastructure and organization, not centralizing it as we are doing. Michael > Well, in this particular case, it was OIT's fault, and it was too slow, > though my experience has frequently shown this not to be true. > The IPS > was set to trigger on perfectly valid and expected traffic loads. By > blaming OIT, however, I would like to point out that our experience with > the vast majority of OIT employees in the resolution of this problem was > extremely positive. I place the majority of the blame for both the > problem and the delay in fixing it on one particular individual. In > all, we experienced web downtime for more than a day on this particular > issue. It doesn't show that way in the ticket because apparently the > engineer responsible doesn't bother to use the help ticket system. > > > -----Original Message----- > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > > Behalf Of TuxGirl > > Sent: Saturday, December 03, 2005 11:56 PM > > To: BYU Unix Users Group > > Subject: Re: [uug] BYU net authentication. > > > > > >>Interesting... How long ago were you blocked for having multiple SSH > >>connections? Were you notified? How long did it take them to block you? > >>I've never heard of that policy, but I don't work in the securities group > >>so... > > This was another situation with the IPS, and where the security group > (is there really a group?) was at fault. The problem was trivial to > reproduce, and the "offending host" could trigger the block instantly. > No, there was no notification involved. The resolution of this problem > also took took several days. As with most of our other issues, we had > an extremely positive experience with the vast majority of the engineers > we worked with. > > With this problem, the IPS threshold was set at 6 ssh connections > initiated within 2 seconds, and was repeatedly being triggered due to > the fact that all the computers in each of our computer labs aggregates > traffic through a single IP. Who woulda thunk that 40 linux boxes > behind a single IP could ever have initiated more than 6 ssh connections > within a short period of time? We also saw the block triggered by a > single host through a number of other perfectly legitimate methods (such > as opening several files in konqueror via fish://). > > At our insistence, the threshold is now much higher, but is still in > effect. We are still waiting to see whether we see problems with the > current setting. So far so good. > > Frank > - -- > Frank Sorenson - KD7TZK > Systems Manager, Computer Science Department > Brigham Young University > [EMAIL PROTECTED] > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > iD8DBQFDkrrdaI0dwg4A47wRAppRAJ9Am0eH0/Aco/tgBWpOKdp1hKSzSQCg8FUZ > OMiTXYPsxsDnDpzlMyRPk7o= > =xTIx > -----END PGP SIGNATURE----- > > -------------------- > BYU Unix Users Group > http://uug.byu.edu/ > > The opinions expressed in this message are the responsibility of their > author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. > ___________________________________________________________________ > List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list -- Michael Torrie <[EMAIL PROTECTED]> -------------------- BYU Unix Users Group http://uug.byu.edu/ The opinions expressed in this message are the responsibility of their author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list
