Re: fiber switch for gig
Are you wanting hardened devices for an outside cabinet install (if it's going outside then you'd better want hardened devices) or is this for an internal environmentally-sound install? What's your definition of long distance? 1800ft, 10km, 20km, 40km, 70, 80, 110? Assuming SMF, do you need simplex or duplex? Have you considered talking to a FTTx vendor? We use Occam here and they make some cost-effective fiber products built with FTTx in mind. I'm not sure how they compare price-wise with products you listed below but it might be worth checking out. Justin Andrew Staples wrote: Speaking of running gig long distances, does anyone on the list have suggestions on a 8 port L2 switch with fiber ports based on personal experience? Lots of 48 port gig switches have 2-4 fiber uplink ports, but this means daisy-chains instead of hub/spoke. Looking for a central switch for a star topography to home fiber runs that is cost effective and works. Considering: DLink DXS-3326GSR NetGear GSM7312 Foundry SX-FI12GM-4 Zyxel GS-4012F I realize not all these switches are IEEE 802.3ae, Clause 49 or IEEE 802.3aq capable. Andrew
Re: rack power question
Dorn Hetzel wrote: Of course, my chemistry is a little rusty, so I'm not sure about the prospects for a non-toxic, non-flammable, non-conductive substance with workable fluid flow and heat transfer properties :) Mineral oil? I'm not sure about the non-flammable part though. Not all oils burn but I'm not sure if mineral oil is one of them. It is used for immersion cooling though. Justin
Re: 10GE router resource
Joel Snyder wrote: Also I'd love to hear recommendatios for budget 10GE routers. The budget router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. If you don't need BGP-ish power, David Newman just published his test of 10GigE switches today in Network World. He was focusing mostly on switching in the enterprise, but he has a variety of other performance metrics and results which may be helpful: http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb The author's specifications eliminated Cisco's 4900M from the competition. That not unexpected though since it was a evaluation of access switches w/ 10G uplinks. The 4900M has 8 on-board 10G interfaces and expansion modules that can carry 8 more (not oversubscribed) or 16 (oversubscribed). It has has GigE support via TwinGig modules in the expansion module bays. It also has a 320Gbps backplane and can handle up to 200k v4 routes. It's an impressive little switch if you need 10G aggregation. It can't handle a full table of course but it still has a lot of use. No MPLS options. It's based on the 4500's Sup 6-E. http://www.cisco.com/en/US/products/ps9310/index.html The base unit starts at $16k. Justin
Re: Customer-facing ACLs
Adrian Chadd wrote: Does anyone have any handy links to actual raw data and papers about this? I'm sure we've all got our own personal datapoints to support automated network probes but I'd prefer to stuff something slightly more concrete and official(!) into the Wiki. SANS ISC might have some useful reports. I see a few links in this article: http://www.incidents.org/diary.html?storyid=4045 Justin
Re: Customer-facing ACLs
Ang Kah Yik wrote: However, considering the number of mobile workers out there who send email via their laptops to corporate SMTP servers, won't blocking outbound SMTP affect them? After all, there are also those who frequently move from place to place so they're going to have to keep changing SMTP servers every time they go to a new place that's on a different ISP. Thanks for joining the discussion. Frankly I'd be surprised to find many corps with an externally-accessible SMTP server that would accept mail on tcp/25. The only way they'd do it is with SMTP AUTH which (hopefully) implies the use of SMTP TLS as well. I know of very few corps that actually do this. Most of the corps I can think of are either running Exchange and utilizing RPC over HTTP, simply point their users to their company's webmail server, or require that their users VPN back to HQ to access their internal MTA. The sites that I can think of with external user-accessible SMTP daemons are entities with highly technical users. They utilize SMTP AUTH, TLS, and the Mail Submission Port on tcp/587. I'm afraid they are in the minority though. The MSP port is the best way to get around the blocks with decent MTAs. Your local MTA's support for other non-standard mechanisms for relaying mail from untrusted networks may also help with this problem (RPC over HTTP). Other than that I don't think there's enough demand for outgoing SMTP from the masses to warrant not blocking it. Redirecting generally takes care of that anyway. Thanks for the input though. All thoughts are welcome. Justin
Re: Customer-facing ACLs
Dave Pooser wrote: I can understand the logic of dropping the port, but theres some additional thought involved when looking at Port 22 - maybe i'm not well-read enough, but the bots I've seen that are doing SSH scans, etc, are not usually on Windows systems. I can figure them working on Linux, MacOS systems - but surely the vast majority of 'vulnerable' hosts are those running OS's coming from our favourite megacorp? Which typically don't come shipped with neither SSH server nor SSH client... ? They typically don't ship with an SMTP server either. Considering that my preferred SSH client for Windows weighs in as a single 412k .exe, I'd imagine that bot designers are just writing their own SSH clients for brute-forcing. Or are simply writing a bot that sens TCP SYNs to port 22 and are reporting those hosts that responds with a SYN ACK back to the CC. Then the CC can direct other compromised hosts with a more complete rootkit (or compromised *nix host) to do brute-force userid/password guessing. Half the Mac users? You think? I know a dozen or so sysadmins who use Macs, and about a hundred users who wouldn't know SSH from PCP; I think that's probably a slightly skewed sample considering I'm a Mac geek who hangs around with Mac geeks, and I'd guess the consumer users are a larger percentage of the real-life population. I'd expect the number of folks who want SSH unblocked to be under 1% of a consumer broadband network, and probably closer to 0.1% or so. And again, it ought to be trivial to let your users unblock the system, either via phone call or via self-service Web page (though in the latter case you'd better use a captcha or something so the bot doesn't automatically unblock itself). Agreed. I don't think the end-user's OS makes them more or less likely to be using SSH unless the OS is a BSD or Linux (then I suspect you'd get a disproportionate # of SSH users compared to the other more simple OSs). Justin
Re: Customer-facing ACLs
Mark Foster wrote: Port 22 outbound? And 23? Telnet and SSH _outbound_ cause that much of a concern? I can only assume it's to stop clients exploited boxen being used to anonymise further telnet/ssh attempts - but have to admit this discussion is the first i've heard of it being done 'en masse'. I don't think there's much to be gained from blocking ingress 22 from customers. I don't see any SSH scans originating from my customers (though there is always the potential). I wouldn't have any issues with blocking outbound telnet though but I can't really justify it either since I don't see a real big problem with that kind of traffic originating on my network either. Now inbound SSH and Telnet (destined for my customers) should be blocked IMHO. Doing a little prodding around our netspace I've found most SSH installs to be of a known vulnerable version or at least an old version yet to have any vulnerabilities found in. Nothing positive could come from letting them get compromised. We would of course offer a way for users to get around the block. Our current approach is to have them sign up for a static IP (another $5/month). The fee keeps everyone from automatically signing up for is as free stuff but still gives the legit users an inexpensive way to get what they need. It'd frustrate me if I jacked into a friends Internet in order to do some legitimate SSH based server administration, I imagine... Agreed but remember that people like you, I and the rest of the readers of NANOG are a teeny tiny minority on the Internet. I could pick a couple thousand of my users at random and not find one that knows what SSH is. Is this not 'reaching' or is there a genuine benefit in blocking these ports as well? I don't think there's much to be gained from blocking telnet SSH from the customers to the Internet. Blocking SMTP in the same direction is critical IMHO. Blocking the same 3 ports back to the customer makes sense to me though. I think there is a real and tangible benefit to the exercise. Justin
Re: Customer-facing ACLs
It varies widely. I see some extremely slow scans (1 SYN every 2-5 minutes). This is what someone on the SANS ISC page mentioned I believe. I've also seen scans last for up to 10 minutes. The consistency of the speeds made me think that perhaps the scanning computer was on a slow link. The worst scans are the ones that last a second or two and hit us with a SYN for every IP in our allocations. That kind of scan and its flood of packets is the one that I don't think I can stop without some sort of QoS. I've seen coordinated scans with everything from 2 to about a dozen different hosts scanning seemingly random IPs across our network. I know it's coordinated though because together they hit every IP but never hit the same IP by more than one scanner. I've seen scans that clearly learn where the accessible SSH daemons are, that then feed this info back to the puppet master so he can command a different compromised host (or hosts) to then handle the attacks. I've also see a scanner first scan our network and then immediately start pounding on the accessible daemons. Finally I've see the scanner stop its scan in mid-stream, pound on an accessible daemon for a while with a pre-defined set of userids and then continue on with the scans. Clearly there's some variation in the scanning methods. Justin Frank Bulk wrote: The last few spam incidents I measured an outflow of about 2 messages per second. Does anyone know how aggressive Telnet and SSH scanning is? Even if it was greater, it's my guess there are many more hosts spewing spam than there are running abusive telnet and SSH scans.
Rogue traffic commonly perceived as noise (was: Scan traffic from 121.8.0.0/16)
Yeah, much of it is noise. However there is a a lot of coordination to much of what I'm seeing. Many of the scans stop at hosts with accessible SSH daemons and pound on them for minutes to hours. Others are more subtle. I'll see one host scan our ranges and pick out the IPs running SSH. Then, a short time later, those specific hosts are directly targeted from a different compromised host implying that there is communication on the back-end about IPs w/ SSH daemons. I tested the theory by disabling SSH on a few of the hosts picked up in earlier mass scans. The targeted attacks are still aimed at those hosts learned in the earlier scan even though their SSH daemons we effectively offline. Some scans are so slow they're barely noticeable (as was reports on the SANS ISC site recently). Even though much of this is simply noise and typical life on the Internet, I have to wonder how much of this noise is actual reconnaissance against SPs and their customers. A certain large SE Asian country's military is widely reported to be performing recon and attacks against IP resources around the globe. How much of what people believe is noise is actually malicious traffic or a prelude to some future event? Frankly the scans on my network have been significantly reduced by being a little more proactive with my monitoring. I've found that network generating SSH scans are also being used for telnet, MS-SQL and SMTP scans. Unfortunately the processes I'm utilizing are very labor intensive and I can't keep doing this forever. I would love to find a tool that could help me automate some of this process and hopefully react faster than I can. While typing this 69.13.181.99 just scanned one of our /19s. The flood of packets was so fast I wouldn't have been able to null route it even if I'd been actively watching the flows. The only way I could have slowed it down would have been to rate-limit SYNs. That leads to a good question for NANOG at large which I'll post separately. Justin Martin Hannigan wrote: Scans are really a dime a dozen and noise that buries good data on real problems. Be careful! On 3/6/08, Justin Shore [EMAIL PROTECTED] wrote: Rich Sena wrote: Anyone seeing anything similar - trying to determine if this is spoofed etc... I haven't picked up any SSH or telnet scans from that network. That's what I'm looking for at the moment. The amount of scans we're getting are quite impressive at times. I wish there was an easy way to automate the care and feeding of my RTBH with this data (and some sanity checks). Justin
Customer-facing ACLs
This question will probably get lost in the Friday afternoon lull but we'll give it a try anyway. What kind of customer-facing filtering do you do (ingress and egress)? This of course is dependent on the type of customer, so lets assume we're talking about an average residential customer. Do you block SYNs destined to your customers? Do you rate-limit SYNs destined for your customers? SYNs on privileged ports? Do you block any customer-facing egress traffic at all? What about ingress? SMTP, NetBIOS, MS-SQL, common proxy ports (3128, 6588)? What ICMP types do you allow or disallow? I'm assuming everyone uses uRPF at all their edges already so that eliminates the need for specific ACEs with ingress/egress network verification checks. Do you filter anything destined to your network infrastructure on your customer-facing edges? Does anyone filter traffic destined to the PE side of a PE-CE link from the outside world? For those of you with cable networks, what all do you block with the CM? We're considering blocking NetBIOS and DHCP server traffic (DHCP server packets are already blocked at the CMTS but this would keep that junk off our infrastructure). For SMTP we permit access to our SMTP servers on tcp/25 to all our broadband users. We also permit our customers with static IPs (residential and business) to send SMTP without restrictions. After those permits we explicitly block tcp/25. This has worked fairly well for us. It sure makes it easy to find infected PCs with spambots. We don't touch tcp/587. For ICMP we permit echo, replies, packet-too-big, and time-exceeded. Everything else gets dropped. Frags are explicitly dropped before any permits. We also block common proxy ports to and from the customers (the to includes ports not always used for proxies). This has been very effective in catching a number of bots that scanned for open Squid proxies or script kiddie junk that used WinGate with the default settings. Is there a BCP for customer-facing ACLs? Justin
Re: Customer-facing ACLs
[EMAIL PROTECTED] wrote: On Fri, 07 Mar 2008 13:55:05 CST, Justin Shore said: I'm assuming everyone uses uRPF at all their edges already so that eliminates the need for specific ACEs with ingress/egress network verification checks. You're new here, aren't you? :) Hopefully optimistic. Don't bum me out going into a weekend... :-) From the looks of my ingress BOGON ACLs on my borders (yes, I'm using ACLs and not null routes for a reason) I'd most people not reading NANOG (and maybe even some of them!) are not doing any ingress filtering on their customer source IPs. Sad Justin
Re: Customer-facing ACLs
Scott Weeks wrote: fire + gasoline = religious argument on this issue that we've had *many* times in the past... ;-) I wore my flame-retardent tidy whiteys today though so I'm prepared. :-) I can understand the problem from both camps. As a tech-savvy user I don't want my provider to filter my Internet (I pay for both halves!). However having spent more time that I care to admit doing customer support and as a SP engineer I recognize the need to protect the masses who can't (or can't be bothered to) do it for themselves. SPs are really damned if we do and damned if we don't. Frankly I would rather do something than nothing. My overhead will increase in all categories if I do nothing. Infected hosts consume a significant portion of my resources. Tackling the problem reduces my overall support costs, increases 99% of my customers' overall satisfaction with our prices and services, but increases my labor costs and will spurn the ire of the 1% of my users who are tech-savvy enough to want/need unfiltered Internet access (or even understand the full implications of it, beyond the they're filtering something that was sent to me! limited thought process). To me there is no question of whether or not you filter traffic for residential broadband customers. The only thing beyond that is whether you as a SP also offer unfiltered packages. I personally thought the old SpeakEasy model was a nice approach. The unfiltered SysAdmin package was the perfect solution in my opinion. With my userbase I can think of only a tiny fraction of the users who would need such an offering. This would provide an acceptable solution for that very small percentage of users that need this kind of access. The other 99% of the users reap the benefits of our filtering. Problem solved IMHO. So that only leaves the question of what ports to block or rate-limit. Minimizing FPs is important. I think one could probably block 90% of the common crap without too much overhead. The remaining 10% would likely require too much labor to be worthwhile unless you were a sizable entity and could justify you RD by the sheer mass of your userbase. Justin
Re: Customer-facing ACLs
Scott Weeks wrote: We need to take this off-line. All long timers are groaning, rolling their eyes and putting this in their kill file. Are the long-timers groaning and ignoring this thread? I certainly hope not. It's threads like these that need the benefit of their experience the most. Perhaps the long-timers could recommend a better destination for queries like these because I have more questions I want to ask (my next being about walled gardens). If they're tired of answering the same threads over and over again, then the query must be common enough to warrant a BCP or at the very least a couple documents in a knowledgebase somewhere. Perhaps my Google-fu isn't what it used to be but I couldn't manage to find any relevant docs online; not even a NANOG presentation. Try convincing your product managers to create a new product just to appease 'sysadmin types'. We're not in the business of alienating any customers. If we can create a bundle that meets a group of potential customers' needs we will. It's just another paragraph on the sales literature that we give our CSRs and a little more work that I'll have to do in configuration. I'm planning on rolling out SOHO and Gamer packages this year. Adding a SysAdmin package wouldn't be much additional work. I predict the adoption rate to be the highest with the Gamer package, followed by the SOHO package and finally the SysAdmin package. I hope this thread isn't destined for an untimely death. I've received a number of off-list queries for summary information because those individuals are also interested in customer-facing ACLs. The information I have to summarize at this point is brief and incomplete. Justin
Re: Scan traffic from 121.8.0.0/16
Rich Sena wrote: Anyone seeing anything similar - trying to determine if this is spoofed etc... I haven't picked up any SSH or telnet scans from that network. That's what I'm looking for at the moment. The amount of scans we're getting are quite impressive at times. I wish there was an easy way to automate the care and feeding of my RTBH with this data (and some sanity checks). Justin
Re: YouTube IP Hijacking
Christopher Morrow wrote: On Sun, Feb 24, 2008 at 8:42 PM, Patrick W. Gilmore [EMAIL PROTECTED] wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard. I know of one tier-1 on that list that made a filtering mistake on one of my own circuits no more than a few months ago. Even some of the biggest players make mistakes. Justin
Re: ISP's who where affected by the misconfiguration: start using IRR and checking your BGP updates (Was: YouTube IP Hijacking)
Jeroen Massar wrote: * PHAS: A Prefix Hijack Alert System http://irl.cs.ucla.edu/papers/originChange.pdf (A live/direct BGP-feed version of this would be neat) Does PHAS still work? I tried to submit a request to subscribe a few weeks ago and never heard back from their automated system. I figured the project was terminated but the site was still up. Justin
Re: Blackholing traffic by ASN
Justin Shore wrote: The ASN I'm referring to is that of the Russian Business Network. A Google search should turn up plenty of info for those that haven't heard of them. Thanks for the replies. They were along the lines of what I was expecting (as-path ACL filtering route-maps). I was wondering if there was some new trick that was easier and more robust. This will work though! I saw that AS40989 fell off the 'Net a while back. That happened once or twice before if memory serves me correctly and they came back a while later in force. We'll see what happens this time. Some of RBN's old netblocks are also no longer in the global tables. I'm not sure what's going on with that but... I'm going to have to do a little more research on their current Inet sources to see if I can locate them. It looks like Wikipedia has a fair amount of information and a large number of links to additional information. http://en.wikipedia.org/wiki/Russian_Business_Network I'm going to have to put a little more effort towards getting my blackhole operational. If anyone has any good links to docs or advice on what not to do I'd love to see them. I've found a great deal of information on the 'Net but lessons learned from those who've already been there done that is always welcome. I hadn't considered what Danny pointed out about the origin AS advertising other routes to create an effective DoS mechanism. That would be a concern and would require a great deal of forethought. Null routing prefixes would probably be the best course of action. Thanks for the insight. Justin
Blackholing traffic by ASN
I'm sure all of us have parts of the Internet that we block for one reason or another. I have existing methods for null routing traffic from annoying hosts and subnets on our border routers today (I'm still working on a network blackhole). However I've never tackled the problem by targeting a bad guy's ASN. What's the best option for null routing traffic by ASN? I could always add another deny statement in my inbound eBGP route-maps to match a new as-path ACL for _BAD-ASN_ to keep from accepting their routes to begin with. Are there any other good tricks that I can employ? I have another question along those same lines. Once I do have my blackhole up and running I can easily funnel hosts or subnets into the blackhole. What about funneling all routes to a particular ASN into the blackhole? Are there any useful tricks here? The ASN I'm referring to is that of the Russian Business Network. A Google search should turn up plenty of info for those that haven't heard of them. Thanks Justin
Level3 in the Midwest is KIA
L3 dropped us at 13:30CST. I've been told that whatever happened took out everything from KC to Wichita to Little Rock to Houston. No word on the cause and no ETA yet. They're handing us 37 routes which is a far cry from the roughly 237,000 we'd normally get. I recognize 3 of the routes too as routes local to the Wichita area. FYI Justin
Re: Level3 in the Midwest is KIA
I've been told that there are 2 issues. One was fiber cut, I believe in the Houston area. The second issue was a card failure, also in the Houston area. Both failures contributed to a loss on a backbone ring that covers a portion of the Midwest. The master ticket number is 2332102 for those who want updates. Justin Justin Shore wrote: L3 dropped us at 13:30CST. I've been told that whatever happened took out everything from KC to Wichita to Little Rock to Houston. No word on the cause and no ETA yet. They're handing us 37 routes which is a far cry from the roughly 237,000 we'd normally get. I recognize 3 of the routes too as routes local to the Wichita area. FYI Justin
Re: RIR filtering Level3
Just to followup with the list, there was a small omission in the filtering of the routes on our peering session. That accounts for the more specific routes we were seeing. L3 made the filtering change on their side and we're back down to within a percent or less of our other BGP peers. It wasn't hurting us; our hardware isn't up against any resource limits; I just happened to notice it and thought I'd take the opportunity to inquire about RIR filtering with the group. Thanks for the quick work on this one, Roy and Kevin. I am still interested in implementing some minimum allocation filtering on our borders. I can't think of any reason to accept anything below the minimum of a /24. Can anyone else? None of the DNS root servers are on anything smaller than a /24 are they? Does anyone have any suggestions for implementing this in a sane manner? I'm assuming matching 0.0.0.0/0 ge 24 would be sufficient unless there are some exceptions like perhaps the root servers. Thanks Justin Justin Shore wrote: Are any other L3 customers seeing the large number of /25 and smaller routes from L3?
RIR filtering Level3
Are any other L3 customers seeing the large number of /25 and smaller routes from L3? I'm seeing almost 2500 of these routes in 4/8, some but not as many in 8/8 and still more in L3's non-US allocations. Looking at the AS paths for a handful of those specific networks I only see them via our L3 connection and not via our other 2 upstreams. I'm seeing paths to the larger aggregate networks via our other upstreams of course; the Oregon and ATT route servers see the same aggregates too. To be more accurate we actually touch L3's acquisition form a year or so ago, Telcove (19094). All of the small routes are originating from L3 though (3356). Best I can tell L3 is aggregating before it advertises to a peer but not before it advertises to a customer. Or, on the otherhand, perhaps L3 is advertising without aggregation to Telcove and Telcove is not aggregating before advertising to us. So, that said, what is everyone else doing to perform sanity checks on their learned routes? Are a good many implementing RIR filtering and dropping everything smaller than a /24? L3 of course isn't the only source of these tiny routes but it's so obvious I saw it and wasn't even looking for it. This would explain why I'm getting so many more routes from L3 too. I'm getting 232k from ATT, 233.5k from Cox and 244k from L3. Thanks Justin
Another DNS blacklist is taken down
I thought ya'll might be interested to hear that yet another DNS blacklist has been taken down out of fear of the DDoS attacks that took down Osirusoft, Monkeys.com, and the OpenRBL. Blackholes.compu.net suffered a joe-job earlier this week. Apparently the joe-jobbing was enough to convince some extremely ignorant mail admins that Compu.net is spamming and blocked mail from compu.net. Compu.net has also seen the effects of DDoS attacks on other DNS blacklist maintainers. They've decided that the risk to their actual business is too great and they are pulling the plug on their DNS blacklist before they come under the gun by spammers. http://groups.google.com/groups?hl=enlr=ie=UTF-8oe=UTF-8selm=3f70e839%241%40dimaggio.newszilla.com Ron Guilmette, maintainer of the Monkeys.com blacklists has posted a farewell from Monkeys.com to news.admin.net-abuse.email. Ron cites the total lack of interest in the attacks by both big network providers and law enforcement authorities as the ultimate reason he's pulling the plug. http://groups.google.com/groups?q=%22Now+retired+from+spam+fighting%22hl=enlr=ie=UTF-8oe=UTF-8selm=vn1lufn8h6r38%40corp.supernews.comrnum=4 It's truely a sad day for spam fighters everywhere. So, my question for NANOG is how does one go about attracting the attention of law enforcement when your network is under attack? How does the target of such an attack get a large network provider who's customers are part of the attack to pay attention? Is media attention the only way to pressure a response from either group? These DDoS attacks have received some attention in mainstream media: http://www.msnbc.com/news/959094.asp?0cv=TB10 http://www.boston.com/news/nation/articles/2003/08/28/saboteurs_hit_spams_blockers Apparently it hasn't been enough. Legal remedies take too long and are cost prohibitive (unless you're the DoJ). Subpoenas and civil lawsuits take months if not years. Relief is needed in days if not hours. Justin
RE: Another DNS blacklist is taken down
On Wed, 24 Sep 2003 [EMAIL PROTECTED] wrote: Perhaps, but it also seems like moving an RBL onto a P2P network would making poisoning the RBL far too easy... That's what I was getting ready to suggest. As it stands now we have at least somewhat of an assurance that the zone we're working with isn't tainted. I only use DNSBLs that offer zone transfers. I only get an AXFR from authorized NSs for that DNSBL. Assuming that NS hasn't been compromised I feel fairly safe in assuming that the data I'm getting is valid. It might not be but I feel that it is. If a P2P system was devised for distributing RBL zones then some for of validation for the distributed zones will have to be created. That would most likely involve a central server. Now you have a server to DDoS again. *sigh* We should just educate spammers with clue-by-fours and make the world a better place. Justin
Re: what to do about joe-jobs?
On Wed, 24 Sep 2003, Stephen L Johnson wrote: Please forgive my ignorance, but what is a joe-job? I dug up some links for you. http://www.spamfaq.net/terminology.shtml#joe_job http://www.techtv.com/news/culture/story/0,24195,3415219,00.html http://catb.org/~esr/jargon/html/J/joe-job.html http://www.everything2.com/index.pl?node=Joe%20Job (might be down?) Basically it's of spoofing the source of spam so as to appear to come from an innocent person. I've been on the receiving end of it a couple of times. Basically the innocent person gets flooded with bounces from poorly written MTAs and anti-spam scripts. Think email-based virus bounces. You didn't send the virus; you aren't even infected. However some machine somewhere is infected and spoofed your address as source of the infected email. You of course end up with the bounce and blame from uneducated people for being infected (which again you are not). Hope that helps Justin
RE: Another DNS blacklist is taken down
On Wed, 24 Sep 2003, Joel Perez wrote: Great, Just Great. Wasn't there a post a while back that listed what providers are SPAM friendly? My fingers are getting tired trying to create ACL's lists to block ranges of IP's without compromising my service. I wish the power's up above would buy the right software to try and curb the SPAM but that is not to be according to them. So back to my ACL's I go! This is one of the most likely things to happen. DNS RBLs are effective. Otherwise spammers wouldn't be targeting them for abuse. Mail admins will eventually start running their own RBLs or rejecting mail by other means locally. This distributed method creates hundreds and eventually thousands of separate points of contact for getting yourself off a RBL. I ran my own domain and netblock list in the past and I can say from experience that it is a very time consuming process. At the time it was also extremely effective. I didn't list open relays/proxies/formmail.cgi IPs. I did however list spamming domains and providers. It caught a surprising amount of spam. It also left me with little time to do anything else. There's got to be a better way. Justin
Re: what to do about joe-jobs?
On Wed, 24 Sep 2003, Stephen J. Wilcox wrote: The one that they're doing on my own domain which I mentioned on list some months ago is still going strong with many Mbs of bounces per day.. I think its fair to say there is very little you can do as tracking the source is almost impossible.. That depends on how detailed the bounce is, to an extent. Many of the bounces actually contain a complete copy of the message that generated the bounce. Ie, the full spam and nothing but the spam. From that you can find the original source IP. Of course that source IP may very well be an open proxy. You're screwed if that's the case. However since you have a complete copy of the spam you can still follow the money trail. Spammers have to get their money somehow. The actual spam will give you many places to start. Of course once you have that you still have to convince a provider to take action against their customer. Justin
Re: williams spamhaus blacklist
On Wed, 24 Sep 2003 [EMAIL PROTECTED] wrote: Customers who use blacklists compiled by vengeance-oriented folk deserve what they get: No email. Suggested solutions: a) whitelist williams b) stop using SBLs similar to spamhaus. It is a question of trust: Do you trust spamhaus to block 'evil' spammers? Do you trust them after they blocked important mails to your clients that could -not- -possibly- have been spam? Make your own conclusions. -alex Providers that sleep the dogs deserve exactly what they get: No email. Suggested solutions: a) find a ethical provider that responses to abuse complaints b) I can't think of anything better than a. It is a question of trust: Do you trust Williams to be ethical to their Internet peers and respond to abuse issues? Do you trust them after they -repeatedly- -ignore- abuse complaints regarding your clients receiving spam from a spamhaus on their network? Make your own conclusions. -justin
Re: Providers removing blocks on port 135?
On Tue, 23 Sep 2003, Mike Tancsa wrote: The credit cards in our case were legit. They were different numbers, but they were not stolen. That would make a difference. The credit card companies probably wouldn't care if you told them that the cards were being used by their customer for illegal activity. Stolen, maybe. Anything else they could probably care less about. They are usuaully fairly responsive when it comes to them loosing money. Secondly after I contacted the local police, state BI, and perhaps the FBI (assuming no luck could be had with any of them) I am in Canada, but I know that it has been stated that the FBI will not investigate computer fraud if damage is under $100,000. I didn't realize you were in Canada. That makes a difference. The dollar amount with the FBI varies widely. I've heard over $5,000, $25,000, $50,000, now $100,000. I don't think there's a hard set rule. I think it basically boils down to the old fashioned will-it-get-us-good-PR-with-little-or-not-work rule of thumb. :) I doubt a porn company with international clientele would give a toss about what the local media say. Local media would have been useful not against the spammers but against the local lw enforcement that refuses to do anything about it. Since however your local law enforncement can't do much about (international borders et al) then the local media wouldn't really care. Local government has nothing to do with it. It was just some dime a dozen porn company. Ditto for what I said above. The part about being in Canada changes things considerably from what I was assuming. I must have missed that earlier. Still it's worked in the past in other circumstances so the practice is fairly sound. It just won't work in your case. My only other advice would involve what's suggested in man 8 syslogd under SECURITY THREATS, option number 5. Best of luck. Justin
Re: Operations notification manager software
On Mon, 22 Sep 2003, Stephane Bortzmeyer wrote: On Mon, Sep 22, 2003 at 12:23:35AM -0500, Justin Shore [EMAIL PROTECTED] wrote a message of 20 lines which said: What software is available/recommended for NOC contact management? I've used Nagios (formerly NetSaint) in the past and have been very impressed with it. I used Nagios and I fail to see what's the connection with the original question? It seems the original poster is looking for something like RequestTracker URL:http://www.bestpractical.com/rt/ instead. From the original message: - contact information refresh (regularly verify contact information via electronic or triggered human interaction, dealing with failed notification attempts)? I need more info here such as an example. Verify contact info against what? - complex notification (ie per-event customized notification by affected device/region/service, notification to customer-selected method based on type and urgency of notice) Nagios - customer-friendly subscription management (including multiple notification methods) and notification archiving Nagios - notification SLA's (ie re-sending multiple timed notices when required, tracking notifications for auditing, etc) Nagios - efficiently managing multiple conduits for notification (email, alpha pager, text-to-voice/scripted call center, RSS feed, Web archives/posting) Nagios - enforcing consistency in notifications (ie form-/ rule-based notification creation and validation, notification review/authorization prior to distribution) I don't know of a way to review/authorize notifications before going out but it wouldn't exactly be hard to script and use with Nagios. - handling feedback from notifications (handling customer responses, tracking viewing and/or reading of notifications, measuring effectiveness of notifications) Nagios doesn't do this. It can accept comments from admins responsible for a given system/service but that's it that I'm aware of. Tracking feedback sounds more like a ticketing system to me. - other important features? Nagios has numerous useful features. One of the most useful features is failure notification esculation. 'An email about an outage sent to the sysadm responsible for the mail system go unanswered (ie the problem still exists and hasn't been acknowledged)? Esculate it. Page the on-call pager and let whoever is on-call call the responsible admin on the telephone.' Very handy feature. IMHO I think Nagios fits most of the specifications the requesting person wants. Justin
Re: Providers removing blocks on port 135?
On Sun, 21 Sep 2003, Mike Tancsa wrote: Yes, this is all too familiar. Luckily it was not so acute for us. The porn company in question was using legit credit cards and we knew where they were located. We too got to the point where I had to contemplate blocking dialups with no ANI as I had already blocked all access from their phone numbers. However, once they started doing that I called up their office yelling and screaming law suits and I guess they figured there were other ISPs that didnt care as much and moved on to them. I don't know if you did this but if it were me I'd have contacted two other places. The first would have been the credit card companies with the stolen credit cards. They are usuaully fairly responsive when it comes to them loosing money. Secondly after I contacted the local police, state BI, and perhaps the FBI (assuming no luck could be had with any of them) I would have given the story to the local media. There's nothing like a little bad PR to give law enforcement a little kick in the butt. If your newspapers where you're at are anything like our's, they love to print a good scandal involving the local government. Justin
Re: Providers removing blocks on port 135?
On Sat, 20 Sep 2003, Margie wrote: Very little spam coming off dialups and other dynamically assigned, residential type connections has anything to do with open relays. The vast majority of it is related to open proxies (which the machine owners do not realize they are running) and machines that have been compromised by various viruses and exploits. These are machines that should not be running outbound mailservers, and in most cases, the owners neither intend nor believe that their systems are sending mail. Merely stating that people shouldn't run open relays didn't stop spam four years ago and it is less likely to do so now. This veers off the original topic. Of course I don't think any of us recall what that was anyways... I remember back when I first started using the DUL. Of all the DNSBLs I used at the time it blocked the most spam of any of them. I mean that by long shot. About the time the DUL and other MAPS lists went commericial is about the same time I noticed fewer and fewer hits on the DUL. We still pay for an AXFR (IXFR) of it but it doesn't block nearly as much as it used to. The open proxy lists block an unbelievable amount of spam. In theory the DUL would take care of this if it also list residential dynamically assigned cable/dsl lines (if it doesn't already, hmmm...). Still the open proxy DNSBLs seem to be more effective now. Bottom line, use every DNSBL you possibly can and don't be afraid to pay for them. I strongly recommend redirecting SMTP traffic for this same class of user as well. Now I'm going to get even more off-topic. It occurs to me that major changes to a protocol such as SMTP getting auth should justify utilizing a different tcp/ip port. Think about it like this. If authenticated forms of SMTP used a different TCP/IP port we netadms could justify leaving that port open on these same dynamically assigned netblocks in the theory that they are only able to connect to other authenticated SMTP services. Doesn't that seem logical? Justin
Re: Providers removing blocks on port 135?
On Sat, 20 Sep 2003, Sean Donelan wrote: It costs service providers more (cpu/ram/equipment) to filter a connection. And even more for every exception. Should service providers charge customers with filtering less (even though it costs more), and customers without filtering more (even though it costs less)? If the unfiltered connection was less expensive, wouldn't everyone just buy that; and we would be right back to the current situation? Abosulutely. At least if the customer wants technical support or plans on paying for their bandwidth. It costs *more* resources for an ISP to *not* filter ports and it costs them *less* resources to filter known ports that are rarely used by Joe Blow average user but the cause of 99% of their (our) headaches. How many people here have ever worked in a helpdesk with hundreds of users calling you for help when they've been infected with the latest greatest Netbios-enabled virus and lost their report, thesis, archived email, pictures of the kids, you name it. I used to work at a Unv helpdesk. Every single time the mail server hiccuped for whatever reason, or the personal webserver was offline for a few minutes of maintenance in the week hours of the morning (no matter whether it was 2 minutes of 2 days) people would inundate us with complaints. All the real problems had to be put on hold so we could answer the phones. Technical support costs an ISP many times that of the neccessary CPU and RAM resources on an access server or border router needed to filter malicious ports. Why don't we just wait until we identify that a user has been infected or compromised (by whatever resource-hog of a method that entails). Then we can just disable their account and wait for them to call. Those calls are always the most pleasant of the day. When did proactive security measures become criminal? Was there a memo I missed? Justin
RE: Providers removing blocks on port 135?
On Fri, 19 Sep 2003, Matthew Kaufman wrote: I agree entirely with this. You shouldn't call yourself an ISP unless you can transport the whole Internet, including those bad Microsoft ports, between the world and your customers. I disagree. In my opinion a NSP shouldn't filter traffic unless one of its customers requests it. However I strongly believe that an ISP (where it's customers are Joe Blow average citizen and Susy Homemaker) should take every reasonable step to protect it's users from malicious traffic and that includes filtering ports. For example I have no reservation about NATing basic dialup users. I also have no problem with filtering ports for services they shouldn't be running on a dialup connection (HTTP, FTP, DNS) or for services that IMHO have no business on the public internet (including every single Microsoft port I can identify). To not do so is IMHO to run a network in an extremely negligent manner. We do this very thing with email. We filter known malicious messages, attachments, and spam from email. I don't think there's a reasonable person among us that can complain about that. Why not do it to network traffic then? If we should allow every bit of traffic to pass unmolested by ACLs then we should allow all email to pass by unmolested by anti-virus and spam checks. It's a two-way street. On the other hand, what's a provider to do when their access hardware can't deal with a pathological set of flows or arp entries? There isn't a good business case to forklift out your DSLAMs and every customer's matching CPE when a couple of ACLs will fix the problem. For that matter, there isn't a very good business case for transporting Nachi's ICMP floods across an international backbone network when you can do a bit of rate-limiting and cut your pipe requirements by 10-20%. A good point. Justin
Re: Worst design decisions?
On Thu, 18 Sep 2003, David Barak wrote: --- Matt [EMAIL PROTECTED] wrote: I've got a couple others in my head from 3Com and a couple of others, but I thought I'd get the ball rolling. So, what do you think? Personally my issues are console-cable related: is there a benefit to the HUGE variety of console pinouts used by the various hardware vendors? Just look at vendor C as an example (I can think of four types immediately) - not only are the types of console port not standardized, but process for determining the location of the port clearly involved the reading of entrails... Applause I can think of 6 different console cable pinouts and connectors that Enterasys (Cabletron) has used over the years. No wait, make that 7. How could I forget the inherited Fore ATM architecture and subsequent blades. Could people just pick ONE pinout and connector and stick with it? Please! Of course I also have a Cisco 675 that I've been unable to use for years simply because I have yet to figure out what ungodly pinout Cisco used in it. Justin
Re: Worst design decisions?
On Thu, 18 Sep 2003, Todd Vierling wrote: On Thu, 18 Sep 2003 [EMAIL PROTECTED] wrote: : Without a question: PS/2 style keyboard and mouse connectors. Impossible : to tell from each other, And this part is somewhat funny, too, because the PS/2 connector layout is capable of having both devices share the same bus (there's two unconnected pins, which some laptops use to provide alternate CLK/DATA signals). If PS/2 mice used the unconnected pins rather than the same CLK/DATA pins as the keyboard, all machines could simply have two connectors using all six pins and you'd be able to plug either device into either socket. In other words it should work like Apple's ADB (Apple Desktop Bus) ports do (did until they moved to USB). I really miss those ports. Justin
Re: Change to .com/.net behavior
On Mon, 15 Sep 2003, Christopher X. Candreva wrote: On Mon, 15 Sep 2003, Vadim Antonov wrote: I'm going to hack my BIND so it'll discard wildcard RRs in TLDs, as a matter of reducing the flood of advertising junk reaching my desktop. Please share your hack ! I've implemented the official ISC Bind hack on every single one of my name servers and am pushing it and the configuration changes out to my customers as a *required* upgrade. Justin
Re: Sabotage not backhoes: More cable cuts
On Sun, 14 Sep 2003, Sean Donelan wrote: Someone climbed a 15-foot tower in Southern Arizona cutting a fiber optic cable used by Broadwing and Tucson Electric Power. This was within five feet of the 138,000-volt power line. The site was also guarded by barbed wire. At least it's just Broadwing. I mean it not like it's anybody important anyways... Justin
RE: Sabotage not backhoes: More cable cuts
Who Broadwing? Alan Ralsky maybe. I've had Broadwing's 3 /14s and some small misc netblocks in my Sendmail reject list for going on two years. I can't think of anything worthwhile I've missed. :) On Wed, 17 Sep 2003, Winslow, Michael wrote: I'm sure they are important to someone -Original Message- From: Justin Shore [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 17, 2003 12:53 PM To: Sean Donelan Cc: [EMAIL PROTECTED] Subject: Re: Sabotage not backhoes: More cable cuts On Sun, 14 Sep 2003, Sean Donelan wrote: Someone climbed a 15-foot tower in Southern Arizona cutting a fiber optic cable used by Broadwing and Tucson Electric Power. This was within five feet of the 138,000-volt power line. The site was also guarded by barbed wire. At least it's just Broadwing. I mean it not like it's anybody important anyways... Justin
Re: Blocking port 135?
On Fri, 1 Aug 2003, Crist Clark wrote: And for this crowd, I should point out that blocking 135/udp blocks DCE-RPC which is used rather heavily by HP OpenView by default. You may hear some shrieks of pain should you chose to block 135/udp. I bidirectionally blocked all NetBIOS ports (tcp and udp) a long time back and have yet to have any problems. In fact I have blocked every single port that's unique to a Microsoft product including the MS SQL ports. I haven't seen any downside to doing that. I also block all Apple AFP ports for the same reasons. For that matter SunRPC is also blocked. Basically I weeded out all the ports that have major security issues and no valid use for my users. Now I'm not a backbone provider but for my many users we have experienced no problems and have avoided numerous security issues because of it. A little preventative maintenance can go a long way. My $.02 Justin
Re: Complaint of the week: Ebay abuse mail (slightly OT)
I submitted ebay.com to rfc-ignorant.org for this RFC violation almost a year ago (which they of course accepted): http://www.rfc-ignorant.org/tools/detail.php?domain=ebay.comsubmitted=1029353643table=abuse Companies like this could simply care less. If you don't run a mail system with customers expecting to receive mail from ebay then I'd recommend blocking ebay.com. That would include their subsidiary, paypal.com, which BTW is also listed on RFCi. At the least I'd score their mail against the RFCi RHSBLs and add a score of 1. Justin On Sun, 3 Aug 2003, David G. Andersen wrote: To add to the eternally annoying list of companies that ignore abuse@ mail... ebay now requires that you fill in their lovely little web form to send them a note. Even if, say, you're trying to let them know about another scam going around that tries to use the machine www.hnstech.co.kr to extract people's credit card information. Has anyone had success in convincing companies that this is just A Bad Idea (ignoring abuse mail), and if so, how did you manage to do it? Sorry for the slightly non-operational content, but I've had it with ebay on this one. -Dave
Re: Network discovery and mapping
On Sun, 22 Jun 2003, Sean Donelan wrote: Its been a few years since I looked at network discovery and mapping tools. Openview/et al did the job, but was always a pain to move all the boxes to the right spots on the resulting maps. Has network discovery and mapping improved for medium-scale wide area networks for ISPs (e.g. 1,000 networks, 100,000 network devices)? I've found lots of discovery tools, but intelligent mapping/layout still seems to be a problem. The usual requirements for SNMP smart discovery, interface/subnet mapping, device identification and connecting the right symbols with the right lines to all the other symbols. Cheops and Cheops-ng might be useful to you. http://www.marko.net/cheops/ http://cheops-ng.sourceforge.net/ Justin
RE: Spam and following the money
On Thu, 19 Jun 2003, Jay Hennigan wrote: On Wed, 18 Jun 2003, Lars Higham wrote: Joe, While I agree with all of your points individually, I would say that only one of them doesn't work for 'following the money'. This one being the pump-and-dump. Everything else involves a sale of some sort - Send those to [EMAIL PROTECTED]. They work quietly and in the background, but they carry an impressive mallet. I prefer to call it a cluestick but whatever floats your boat will work. :) I especially like option number 5 under SECURITY THREATS in man 8 syslogd. Additional reporting addresses: FDA [EMAIL PROTECTED] Send all food and drug (pharmacueticals) complaints here. ie, weight-loss products, sexual stimulants and enhancements, etc.. FTC [EMAIL PROTECTED] Bounce *all* spam here. Period. SEC [EMAIL PROTECTED] Send all stock scams here. Pyramid/Ponzi scams [EMAIL PROTECTED], [EMAIL PROTECTED] Send these jewels to these folks if they use the USPS. Nigerian Money Scams/419 Scams [EMAIL PROTECTED] Send these to the US Secret Service. Also, if you're feeling bored and want to give a 419 scammer the run around, respond to them with your telephone number ((202) 406-5850) and fax number ((202) 406-5031). Tell them how overjoyed you are to be helping them with their financial problems. Take note that the numbers above are for the United States Secret Service, Financial Crimes Division. :) It my understanding that some people send their 419 complaints to the Nigerian Police at these addresses: [EMAIL PROTECTED], [EMAIL PROTECTED] It was my understanding that the many actual members of the Nigerian government are involved in these scams so YMMV and of course I could be wrong. Justin
Re: OT: question re. the Volume of unwanted email (fwd)
On Wed, 18 Jun 2003, Miles Fidelman wrote: It occurs to me that a lot of people on this list might have that sort of quantitative data - so... any comments? You might find this useful. http://zebulon.miester.org/spam/ Justin
Re: Net-24 top prefix generating bogus RFC-1918 queries
On Sat, 31 May 2003, John Brown wrote: Why does 65/8 generate almost as many queries as 24/8? because there are lots of cable and DSL users in those prefix's My cable at home is net-65 My SBC DSL that this email is coming from is in 65. Justin
Re: dnsbl's? - an informal survey
On Sat, 31 May 2003 [EMAIL PROTECTED] wrote: On Sat, 31 May 2003, Mr. James W. Laferriere wrote: White listing comes with any blacklist. The blacklists in particular being discussed were the @dynamics, like the PDL and dynablock at easynet. Both lists quite clearly state how they build their lists and what they are designed to block (dynablock only takes out dialup, and PDL takes out all dynamic addressing). Query , How is it determined that the address in question is dynamic or not ? Who/how/what makes that determination ? This is the core of my concerns . It's usually determined via in-addr.arpa, whois data, or direct information from the provider. When MAPS was freely available, I used to periodically email them updates on our IP space (please add these dial ranges, please remove these others). I'm sure others did the same. AFAIK, they had at least one FTE who's job it was to maintain the DUL. Many providers list their own dynamically assigned blocks voluntarily. It helps the fight against spam to an extent; plus it's good PR. Someday I expect to either see someone create a list of known MTAs through which you must register it with some entity, or a list of everything that isn't an MTA--every statically/dynamically assigned desktop, laptop, home node, etc... If that ever happens the results should be quite interesting. Those large providers who stole copies of the DUL before MAPS pulled the plug on them, and continued to use them without maintenance still annoy me as we've run into issues multiple times with space removed from the DUL still being in their private copies. I agree. Something like that could have large chunks go stale in a hurry. If you toss in the number of providers going belly-up since MAPS went commercial, then that's a lot netblocks that shouldn't be in the DUL and aren't if people are paying for a current copy (like we do). Justin
