network or not? Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, 15 Mar 2004, Alexei Roudnev wrote: First, let me say that I appreciate your s wrt the s2n ratio here. I don't want to indicate otherwise. But, to get into the circle with everyone else and shoot some marbles... :) : Ok - is name resoluution issue network issue or not? if it is, how can you : answer anything without knowing, for example, of existing Windows DNS : client with internal cache, and difference between 'ping' and 'nslookup' : name resolution on Solaris? : : Is ARP problem - network one or not? if it is, how can you determine, what : happen, if some crazy server became ARP proxy and sends wrong : information to everyone? Loopback plug, sniffer or some similar geek thingie. Not the network; hand the ticket off. I guess it means defining what we mean by the network. : For tier-2 - I agree. For real tier-3 - I can not. Those friends, who are : excellent network engineers (much better than me, with CCIE : and other _really good_ experience), knows Windows and Unix on a very good : level. (of course, if some HR asks them 'where is configuration file for : SAMBA on Solaris - no one answer, but it does not mean that they do not know : Solaris; and you can always met religious people 'my god is MS / my god is : Linux'). I never said a good netgeek didn't know these things. I only said, you don't HAVE to know them to be a good escalation network engineer for a big ass network with specialized folks. : Is it bad, If they (your sysadmins) understand your backbone : infrastructure and understand such things, as MTU MTU discovery, knows : about ACL filters (without extra details) and existing limitations? They : are not required to know about VPN mode or T3 card configuration, but : they must understand basic things. This is what makes good network/system engineers on both sides of the fence. When the ticket is tossed over the fence, the crapwork is done. Person that gets the ticket is happy and returns the favor when tossing a ticket your way. Get both sides caring about tossing tickets properly and you gotta kick-ass team going on. damn, i miss the days... : Else, everything ends up in a long delays and 10 person technical : meetings (by the phone, of course) - which is the best way of wasting : anyone's time. OUCH!!! The pain in my brain from absorbing that idea!! :-) scott : : - Original Message - : From: Scott Weeks [EMAIL PROTECTED] : To: [EMAIL PROTECTED] : Sent: Monday, March 15, 2004 1:32 PM : Subject: Re: Platinum accounts for the Internet (was Re: who offers cheap : (personal) 1U colo?) : : : : : : On Mon, 15 Mar 2004, Alexei Roudnev wrote: : : : I expect, that good (tier-3, to say) network engineer MUST know Windows : and : : Unix (== Linux, FreeBSD etc) on tear-2 (or better) level. Else, he will : not : : be able to troubleshout his _network problem_ (because they are more : likely : : complex Network + System + Application + Cable problem). : : : : So, it is not a good answer. : : No true in many cases. All I have to prove is it's not the network and : then I hand it off to the windows/*nix/whatever sysadmins. To prove : it's not the network, I don't need to know the end systems in any sort of : detail. : : scott : : : : : : : - Original Message - : : From: Pete Templin [EMAIL PROTECTED] : : To: [EMAIL PROTECTED] : : Sent: Monday, March 15, 2004 7:16 AM : : Subject: Re: Platinum accounts for the Internet (was Re: who offers : cheap : : (personal) 1U colo?) : : : : : : : : Laurence F. Sheldon, Jr. wrote: : : : : Pete Templin wrote: : : There's a reason I've gotten out of small ISP consulting - I don't : do : : Windows, and I'm getting overrun by Linux corrosion slowly. I : route, : : I switch, I help with securing networks. And I do wear a lot of : hats : : at my day job, but I remind them that they hired a specialist, and : : promised lots of server support all along the way. Granted, the : : Windows guy is overloaded and the UNIX/Linux guy would snore in : front : : of his PHB... : : : : If you are in Nebraska I can help you with the Unemploy^WWorkforce : : Development paperwork. : : : : I didn't suggest saying I'm not gonna do it. I just suggested You : : hired me to deploy dynamic routing on your statically-routed network. : : What prompted you to think that I could configure site-wide anti-virus : : services such that no one ever reports a virus leak from our : enterprise, : : without training, time to test and develop such a critical solution, : or : : both? : : : : pt : : : : : : :
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
I expect every NANOG conference from now on will be filled with announcements asking people to please fix their computers because worms are killing the network. NANOG has less than 500 attendees, yet has about the same number as infected computers as any other ad-hoc network population. Maybe NANOG needs to implement a system where you have to log in to a web page with your NANOG meeting passcode in order to get a usable IP address. Then, when an infected computer shows up we will know exactly whose it was. Might even be interesting for a researcher to interview every infected party and figure out why it is happening even among a supposedly clueful group. --Michael Dillon
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
[EMAIL PROTECTED] writes: Maybe NANOG needs to implement a system where you have to log in to a web page with your NANOG meeting passcode in order to get a usable IP address. Then, when an infected computer shows up we will know exactly whose it was. Might even be interesting for a researcher to interview every infected party and figure out why it is happening even among a supposedly clueful group. Seconded. This is dirt simple to do. If we believe in public humiliation, a list of infected machines and their owners (along with a suitably snarky don't hire these top network engineers to maintain your fleet of windows boxes message) could be displayed on the projection screens at the break. ---Rob
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
a suitably snarky don't hire these top network engineers to maintain your fleet of windows boxes message) could be displayed on the Is this an opt-in list? I'd like to opt-in. Now. Nu. Proto. A lifetime ago.
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Robert E. Seastrom wrote: Seconded. This is dirt simple to do. If we believe in public humiliation, a list of infected machines and their owners (along with a suitably snarky don't hire these top network engineers to maintain your fleet of windows boxes message) could be displayed on the projection screens at the break. Employee to PHB: You hired me to provide core network engineering and lead the level 2 network ops staff. Tell me again why you want me to provide any server engineering, if you knew my strengths when you hired me? There's a reason I've gotten out of small ISP consulting - I don't do Windows, and I'm getting overrun by Linux corrosion slowly. I route, I switch, I help with securing networks. And I do wear a lot of hats at my day job, but I remind them that they hired a specialist, and promised lots of server support all along the way. Granted, the Windows guy is overloaded and the UNIX/Linux guy would snore in front of his PHB... pt
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Pete Templin wrote: Employee to PHB: You hired me to provide core network engineering and lead the level 2 network ops staff. Tell me again why you want me to provide any server engineering, if you knew my strengths when you hired me? There's a reason I've gotten out of small ISP consulting - I don't do Windows, and I'm getting overrun by Linux corrosion slowly. I route, I switch, I help with securing networks. And I do wear a lot of hats at my day job, but I remind them that they hired a specialist, and promised lots of server support all along the way. Granted, the Windows guy is overloaded and the UNIX/Linux guy would snore in front of his PHB... If you are in Nebraska I can help you with the Unemploy^WWorkforce Development paperwork. -- Requiescas in pace o email
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Laurence F. Sheldon, Jr. wrote: Pete Templin wrote: There's a reason I've gotten out of small ISP consulting - I don't do Windows, and I'm getting overrun by Linux corrosion slowly. I route, I switch, I help with securing networks. And I do wear a lot of hats at my day job, but I remind them that they hired a specialist, and promised lots of server support all along the way. Granted, the Windows guy is overloaded and the UNIX/Linux guy would snore in front of his PHB... If you are in Nebraska I can help you with the Unemploy^WWorkforce Development paperwork. I didn't suggest saying I'm not gonna do it. I just suggested You hired me to deploy dynamic routing on your statically-routed network. What prompted you to think that I could configure site-wide anti-virus services such that no one ever reports a virus leak from our enterprise, without training, time to test and develop such a critical solution, or both? pt
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Pete Templin wrote: Laurence F. Sheldon, Jr. wrote: Pete Templin wrote: There's a reason I've gotten out of small ISP consulting - I don't do Windows, and I'm getting overrun by Linux corrosion slowly. I route, I switch, I help with securing networks. And I do wear a lot of hats at my day job, but I remind them that they hired a specialist, and promised lots of server support all along the way. Granted, the Windows guy is overloaded and the UNIX/Linux guy would snore in front of his PHB... If you are in Nebraska I can help you with the Unemploy^WWorkforce Development paperwork. I didn't suggest saying I'm not gonna do it. I just suggested You hired me to deploy dynamic routing on your statically-routed network. What prompted you to think that I could configure site-wide anti-virus services such that no one ever reports a virus leak from our enterprise, without training, time to test and develop such a critical solution, or both? It turns out that they can hire people with all kinds of certifications that say thye can do all of that for a lot less than what they are paying a specialist. -- Requiescas in pace o email
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, 15 Mar 2004 04:57:03 -0500 (EST), Sean Donelan wrote: NANOG has less than 500 attendees, yet has about the same number as infected computers as any other ad-hoc network population. If true this is a very significant fact
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On 15 Mar 2004 08:01:15 -0500 Robert E. Seastrom [EMAIL PROTECTED] wrote: Maybe NANOG needs to implement a system where you have to log in to a web page with your NANOG meeting passcode in order to get a usable IP address. Then, when an infected computer shows [...] Seconded. This is dirt simple to do. If we believe in public humiliation, a list of infected machines and their owners (along with [...] In the case of some networks and some type of malware, you might need to do more than this. For example, if a compromised host continues to spew out packets without a valid IP, this still eats link capacity. If the network is relatively flat, which is often is in wireless configurations, you still have a problem to solve before normal access for everyone else is restored. John
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
John, There are the beginnings of some wireless devices that are capable of directing wireless clients to cease transmission with L2 link control messages. These are just beginning to emerge, and unfortunately I'm certain that with only a matter of time people will write drivers that ignore such control messages. The end result is that AP's can effectively address a DoS at an invalid/penalty-boxed host on the wireless ether, and allow everyone else to remain connected. There is a b/w penalty for the flood of control messages. One implementation I have been researching leaves ~75% of b/w available for valid traffic. That doesn't seem too bad to me, but I need to research real stats for how much b/w is consumed by the worms in the first place. Cheers, Ben. John On 15 Mar 2004 08:01:15 -0500 John Robert E. Seastrom [EMAIL PROTECTED] wrote: Maybe NANOG needs to implement a system where you have to log in to a web page with your NANOG meeting passcode in order to get a usable IP address. Then, when an infected computer shows John [...] Seconded. This is dirt simple to do. If we believe in public humiliation, a list of infected machines and their owners (along with John [...] John In the case of some networks and some type of malware, you might need to John do more than this. For example, if a compromised host continues to spew John out packets without a valid IP, this still eats link capacity. If the John network is relatively flat, which is often is in wireless configurations, John you still have a problem to solve before normal access for everyone else John is restored. John John
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Laurence F. Sheldon, Jr. wrote: Pete Templin wrote: I didn't suggest saying I'm not gonna do it. I just suggested You hired me to deploy dynamic routing on your statically-routed network. What prompted you to think that I could configure site-wide anti-virus services such that no one ever reports a virus leak from our enterprise, without training, time to test and develop such a critical solution, or both? It turns out that they can hire people with all kinds of certifications that say thye can do all of that for a lot less than what they are paying a specialist. You're right again. But those generalists would earn a spot on the don't hire these top network engineers to maintain your fleet of windows boxes list projected on the screen, while the specialists either wouldn't be doing work outside their scope or the PHB would understand that it's not their specialty. pt
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
I expect, that good (tier-3, to say) network engineer MUST know Windows and Unix (== Linux, FreeBSD etc) on tear-2 (or better) level. Else, he will not be able to troubleshout his _network problem_ (because they are more likely complex Network + System + Application + Cable problem). So, it is not a good answer. - Original Message - From: Pete Templin [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, March 15, 2004 7:16 AM Subject: Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?) Laurence F. Sheldon, Jr. wrote: Pete Templin wrote: There's a reason I've gotten out of small ISP consulting - I don't do Windows, and I'm getting overrun by Linux corrosion slowly. I route, I switch, I help with securing networks. And I do wear a lot of hats at my day job, but I remind them that they hired a specialist, and promised lots of server support all along the way. Granted, the Windows guy is overloaded and the UNIX/Linux guy would snore in front of his PHB... If you are in Nebraska I can help you with the Unemploy^WWorkforce Development paperwork. I didn't suggest saying I'm not gonna do it. I just suggested You hired me to deploy dynamic routing on your statically-routed network. What prompted you to think that I could configure site-wide anti-virus services such that no one ever reports a virus leak from our enterprise, without training, time to test and develop such a critical solution, or both? pt
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, 15 Mar 2004, Alexei Roudnev wrote: : I expect, that good (tier-3, to say) network engineer MUST know Windows and : Unix (== Linux, FreeBSD etc) on tear-2 (or better) level. Else, he will not : be able to troubleshout his _network problem_ (because they are more likely : complex Network + System + Application + Cable problem). : : So, it is not a good answer. No true in many cases. All I have to prove is it's not the network and then I hand it off to the windows/*nix/whatever sysadmins. To prove it's not the network, I don't need to know the end systems in any sort of detail. scott : : - Original Message - : From: Pete Templin [EMAIL PROTECTED] : To: [EMAIL PROTECTED] : Sent: Monday, March 15, 2004 7:16 AM : Subject: Re: Platinum accounts for the Internet (was Re: who offers cheap : (personal) 1U colo?) : : : : Laurence F. Sheldon, Jr. wrote: : : Pete Templin wrote: : There's a reason I've gotten out of small ISP consulting - I don't do : Windows, and I'm getting overrun by Linux corrosion slowly. I route, : I switch, I help with securing networks. And I do wear a lot of hats : at my day job, but I remind them that they hired a specialist, and : promised lots of server support all along the way. Granted, the : Windows guy is overloaded and the UNIX/Linux guy would snore in front : of his PHB... : : If you are in Nebraska I can help you with the Unemploy^WWorkforce : Development paperwork. : : I didn't suggest saying I'm not gonna do it. I just suggested You : hired me to deploy dynamic routing on your statically-routed network. : What prompted you to think that I could configure site-wide anti-virus : services such that no one ever reports a virus leak from our enterprise, : without training, time to test and develop such a critical solution, or : both? : : pt : :
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
No true in many cases. All I have to prove is it's not the network and then I hand it off to the windows/*nix/whatever sysadmins. To prove it's not the network, I don't need to know the end systems in any sort of detail. to pass the buck, one needs to know nothing. what makes a great noc engineer is taking ownership of the user's problem. randy
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, Mar 15, 2004 at 12:21:54PM -1000, Randy Bush wrote: No true in many cases. All I have to prove is it's not the network and then I hand it off to the windows/*nix/whatever sysadmins. To prove it's not the network, I don't need to know the end systems in any sort of detail. to pass the buck, one needs to know nothing. what makes a great noc engineer is taking ownership of the user's problem. The fact of the matter is, business environments today do not frequently seek specific expertise to solve specific problems, preferring instead to (ab)use existing employees to do more than they were hired to do with less time, less training, and fewer resources than they need. Similarly, experts brought in from the outside are usually expected to opine on their areas of expertise as little as possible so that they can be similarly (ab)used to do things other than what they were contracted to do. While taking responsibility for solving problems is an important quality, knowing how to effectively use your time is equally important. On a good note, contract killers seem exempt from this trend. Kelly
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, 15 Mar 2004, Randy Bush wrote: : No true in many cases. All I have to prove is it's not the network and : then I hand it off to the windows/*nix/whatever sysadmins. To prove : it's not the network, I don't need to know the end systems in any sort of : detail. : : to pass the buck, one needs to know nothing. what makes a great noc : engineer is taking ownership of the user's problem. In smaller networks, sure. However, it's not about passing the buck in large networks. It's about responsibilities. There, if you take ownership of the sysadmin's part of the ticket (where there're a lot of sysadmins for every OS), you'll likely get =them= chopped off and hung on the wall as an example to others. I would be pissed if one of the sysadmin folks tried to troubleshoot the backbone network instead of handing it off to me after clearing their part of the problem... All I need to do is clear my part and pass it to them with all helpful data points included in the ticket. Any more than that and I'm stomping on other folks' toes. scott
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
On Mon, 15 Mar 2004 [EMAIL PROTECTED] wrote: Maybe NANOG needs to implement a system where you have to log in to a web page with your NANOG meeting passcode in order to get a usable IP address. Then, when an infected computer shows up we will know exactly whose it was. Might even be interesting for a researcher to interview every infected party and figure out why it is happening even among a supposedly clueful group. I find it ironic that one of the presentations at the last nanog was about a system kind of like that: http://www.nanog.org/mtg-0402/gauthier.html and that we had some luser on the nanog30 wireless network infected by SQL slammer. Does anyone know who that was, how/if they were located and removed from the network, and whether they brought an infected PC (either via stupidity or as a joke) or simply brought an unpatched system out from behind their firewall/packet filters and got infected before they got a chance to actually use the network? After that incident, I sniffed the wireless for a little while and noticed slammer is alive and well out on the internet and still trying to infect the rest of the internet. We're still blocking it at our transit borders. The one time it was removed (accidentally), a colo customer was infected very shortly after the filter's protection was lost. -- Jon Lewis [EMAIL PROTECTED]| I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
I find it ironic that one of the presentations at the last nanog was about a system kind of like that: http://www.nanog.org/mtg-0402/gauthier.html and that we had some luser on the nanog30 wireless network infected by SQL slammer. Well it wouldnt be nanog without a few infections, password grabs and other random security breaches Does anyone know who that was, how/if they were located and removed from the network, and whether they brought an infected PC (either via stupidity or as a joke) or simply brought an unpatched system out from behind their firewall/packet filters and got infected before they got a chance to actually use the network? Probably genuine error (clueless/oversight), no names.. where is Randy when you want him? After that incident, I sniffed the wireless for a little while and noticed slammer is alive and well out on the internet and still trying to infect the rest of the internet. *jlewis in network sniffing shock!* We're still blocking it at our transit borders. The one time it was removed (accidentally), a colo customer was infected very shortly after the filter's protection was lost. yeah theres lots, we filter for several known worms on the gateway routers at the meetings we sponsor, i recommend nanog sponsors do the same (altho it cant save u from the devil within) Steve -- Jon Lewis [EMAIL PROTECTED]| I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Ok - is name resoluution issue network issue or not? if it is, how can you answer anything without knowing, for example, of existing Windows DNS client with internal cache, and difference between 'ping' and 'nslookup' name resolution on Solaris? Is ARP problem - network one or not? if it is, how can you determine, what happen, if some crazy server became ARP proxy and sends wrong information to everyone? For tier-2 - I agree. For real tier-3 - I can not. Those friends, who are excellent network engineers (much better than me, with CCIE and other _really good_ experience), knows Windows and Unix on a very good level. (of course, if some HR asks them 'where is configuration file for SAMBA on Solaris - no one answer, but it does not mean that they do not know Solaris; and you can always met religious people 'my god is MS / my god is Linux'). - Original Message - From: Scott Weeks [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, March 15, 2004 1:32 PM Subject: Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?) On Mon, 15 Mar 2004, Alexei Roudnev wrote: : I expect, that good (tier-3, to say) network engineer MUST know Windows and : Unix (== Linux, FreeBSD etc) on tear-2 (or better) level. Else, he will not : be able to troubleshout his _network problem_ (because they are more likely : complex Network + System + Application + Cable problem). : : So, it is not a good answer. No true in many cases. All I have to prove is it's not the network and then I hand it off to the windows/*nix/whatever sysadmins. To prove it's not the network, I don't need to know the end systems in any sort of detail. scott : : - Original Message - : From: Pete Templin [EMAIL PROTECTED] : To: [EMAIL PROTECTED] : Sent: Monday, March 15, 2004 7:16 AM : Subject: Re: Platinum accounts for the Internet (was Re: who offers cheap : (personal) 1U colo?) : : : : Laurence F. Sheldon, Jr. wrote: : : Pete Templin wrote: : There's a reason I've gotten out of small ISP consulting - I don't do : Windows, and I'm getting overrun by Linux corrosion slowly. I route, : I switch, I help with securing networks. And I do wear a lot of hats : at my day job, but I remind them that they hired a specialist, and : promised lots of server support all along the way. Granted, the : Windows guy is overloaded and the UNIX/Linux guy would snore in front : of his PHB... : : If you are in Nebraska I can help you with the Unemploy^WWorkforce : Development paperwork. : : I didn't suggest saying I'm not gonna do it. I just suggested You : hired me to deploy dynamic routing on your statically-routed network. : What prompted you to think that I could configure site-wide anti-virus : services such that no one ever reports a virus leak from our enterprise, : without training, time to test and develop such a critical solution, or : both? : : pt : :
Re: Platinum accounts for the Internet (was Re: who offers cheap (personal) 1U colo?)
Is it bad, If they (your sysadmins) understand your backbone infrastructure and understand such things, as MTU MTU discovery, knows about ACL filters (without extra details) and existing limitations? They are not required to know about VPN mode or T3 card configuration, but they must understand basic things. Else, everything ends up in a long delays and 10 person technical meetings (by the phone, of course) - which is the best way of wasting anyone's time. : : to pass the buck, one needs to know nothing. what makes a great noc : engineer is taking ownership of the user's problem. In smaller networks, sure. However, it's not about passing the buck in large networks. It's about responsibilities. There, if you take ownership of the sysadmin's part of the ticket (where there're a lot of sysadmins for every OS), you'll likely get =them= chopped off and hung on the wall as an example to others. I would be pissed if one of the sysadmin folks tried to troubleshoot the backbone network instead of handing it off to me after clearing their part of the problem... All I need to do is clear my part and pass it to them with all helpful data points included in the ticket. Any more than that and I'm stomping on other folks' toes. scott
