[sniffer] Re: Appriver issue
For those of us in the dark about this, can someone explain who Appriver is, and what is has to do with Message Sniffer? Thank you, Michael Stein Computer House - Original Message - From: "Kevin Rogers" <[EMAIL PROTECTED]> To: "Message Sniffer Community" Sent: Friday, May 18, 2007 6:45 AM Subject: [sniffer] Re: Downloads are not working I sent a message earlier to this list but I'm not sure if it went through. We've been hit by this Appriver issue and it is still going on as far as I can tell. One of our users, call him [EMAIL PROTECTED] sent a message to about 70 people. And this message has been bounced 20 or 30,000 times and counting. At first I thought it was this Exchange issue we experienced last year where a single message was sent over and over. But then I saw that all the headers of the bounced emails contained calls to appriver.com and when I checked here I found this thread. In the end, the only thing I could do was completely remove that user's account and it appears to be OK. But who knows? Things appeared to be OK from 7pm until 1am PST when it all started up again. Anyone have any information on this? Thanks Kevin Pete McNeil wrote: > Hello Matt, > > Thursday, May 17, 2007, 2:22:56 PM, you wrote: > > >> Appriver, who is somehow involved with Sniffer, is having a ridicolous >> problem with sending messages over and over again (once every few >> seconds). They pulled their contact information from their site but >> didn't take down their servers. I suspect this is putting strain on >> them and if Sniffer uses their bandwidth for downloads, that could >> explain things. >> > > I'm not sure what the actual issue is (I will get that data later), > however I've just been informed that it should be resolved in the next > 20 minutes or so. > > Our rulebase server is on the same network so it is effected. > > BTW - they did not take down their contact information. It is right > where it always has been. > > _M > > # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Hello Computer, Friday, May 18, 2007, 8:34:27 AM, you wrote: > For those of us in the dark about this, can someone explain who Appriver is, > and what is has to do with Message Sniffer? Message Sniffer started out as an incubator project inside of MicroNeil Research Corporation. When it was time for SNF to grow up and become it's own company we went looking for partners who could help it grow. We searched long and hard and ultimately picked AppRiver for a lot of good reasons :-) (Note I can still say that after all this time!) ARM Research Labs, LLC was formed to take over the SNF project. ARM is jointly owned by both AppRiver and MicroNeil. In order to capitalize on economies of scale, asymmetrical bandwidth utilization, and an efficient support infrastructure, a number of servers for ARM are hosted along side AppRiver servers - this includes the servers that provide rulebase updates for SNF. Yesterday's events created a tremendous load on the network and that caused some packet loss that caused rulebase downloads to slow down for a time. For more about ARM see: http://www.armresearch.com/ Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Pete - Thanks for the reply, but I guess I don't understand what you're saying. "Some packet loss" and "rulebase downloads to slow down for a time" don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem "with sending messages over and over again". I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. Kevin Pete McNeil wrote: Hello Computer, Friday, May 18, 2007, 8:34:27 AM, you wrote: For those of us in the dark about this, can someone explain who Appriver is, and what is has to do with Message Sniffer? Message Sniffer started out as an incubator project inside of MicroNeil Research Corporation. When it was time for SNF to grow up and become it's own company we went looking for partners who could help it grow. We searched long and hard and ultimately picked AppRiver for a lot of good reasons :-) (Note I can still say that after all this time!) ARM Research Labs, LLC was formed to take over the SNF project. ARM is jointly owned by both AppRiver and MicroNeil. In order to capitalize on economies of scale, asymmetrical bandwidth utilization, and an efficient support infrastructure, a number of servers for ARM are hosted along side AppRiver servers - this includes the servers that provide rulebase updates for SNF. Yesterday's events created a tremendous load on the network and that caused some packet loss that caused rulebase downloads to slow down for a time. For more about ARM see: http://www.armresearch.com/ Hope this helps, Thanks, _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: > Pete - Thanks for the reply, but I guess I don't understand what you're > saying. "Some packet loss" and "rulebase downloads to slow down for a > time" don't reflect what happened to me yesterday and apparently not > what happened to one of the other posters either when he said that > Appriver was having a problem "with sending messages over and over > again". I received over (at last count) 35,000 messages (almost all of > which were bounced replies, from one email from one of our users who > sent an email to about 70 people) yesterday. > And I had already gone to http://www.armresearch.com/ yesterday and > there was nothing there. There is nothing there today that I can see. > What happened? I lost an entire day's worth of email because of bounced > messages. I didn't sleep last night. I don't even use Appriver. I > would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Thanks for the explanation, and I wasn't trying to blame you - just wanted more info is all. We use Sniffer, but not Appriver. You said that if we don't use Appriver, we shouldn't have been affected, but you also seemed to say that if one of the recipient's of my user's email uses Appriver that might've caused a problem. And also that *some* of Sniffer users might have experienced the problem as well. It sounds like things are still being worked out. I just wanted some kind of verification that they were aware of the problem, were working on it, that they were in some way sorry about what happened...you know - the usual stuff. And I know that you are not an official rep of Appriver or anything, but presently you're all we have in that role ;) Thanks Kevin Pete McNeil wrote: Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. "Some packet loss" and "rulebase downloads to slow down for a time" don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem "with sending messages over and over again". I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Maybe I caused the confusion. The problem I had was with my customer using appriver. Not with my customers using message sniffer. How can something that happens with rulebase downloads effect your mail server? It shouldn't. I would expect there's a seperate problem with your mail server that jus happened to occur the same day by coincidence I received a call from appriver today explaining that they released a patch that had acted badly on their servers. Which is why appriver customers had problems. Message sniffer resides on your own server so it should never be effected by any outside outages Thank You, Chris Bunting Lancaster Networks 717-278-6639 >Sent by my BlackBerry wireless device -Original Message- From: Pete McNeil <[EMAIL PROTECTED]> Date: Fri, 18 May 2007 21:44:18 To:"Message Sniffer Community" Subject: [sniffer] Re: Appriver issue Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: > Pete - Thanks for the reply, but I guess I don't understand what you're > saying. "Some packet loss" and "rulebase downloads to slow down for a > time" don't reflect what happened to me yesterday and apparently not > what happened to one of the other posters either when he said that > Appriver was having a problem "with sending messages over and over > again". I received over (at last count) 35,000 messages (almost all of > which were bounced replies, from one email from one of our users who > sent an email to about 70 people) yesterday. > And I had already gone to http://www.armresearch.com/ yesterday and > there was nothing there. There is nothing there today that I can see. > What happened? I lost an entire day's worth of email because of bounced > messages. I didn't sleep last night. I don't even use Appriver. I > would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
I think what Peter is try to say is that Sort monster is hosted at Appriver and Appriver had an issue and therefore so did Sort monster. http://www.dnsstuff.com/tools/dnsreport.ch?&domain=sortmonster.com Regards David Moore [EMAIL PROTECTED] J.P. MCP, MCSE, MCSE + INTERNET, CNE. www.adsldirect.com.au for ADSL and Internet www.romtech.com.au for PC sales Office Phone: (+612) 9453 1990 Fax Phone: (+612) 9453 1880 Mobile Phone: +614 18 282 648 POSTAL ADDRESS: PO BOX 190 BELROSE NSW 2085 AUSTRALIA. - This email message is only intended for the addressee(s) and contains information that may be confidential, legally privileged and/or copyright. If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email, or taking any action in reliance on its contents by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Rogers Sent: Saturday, 19 May 2007 11:59 AM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue Thanks for the explanation, and I wasn't trying to blame you - just wanted more info is all. We use Sniffer, but not Appriver. You said that if we don't use Appriver, we shouldn't have been affected, but you also seemed to say that if one of the recipient's of my user's email uses Appriver that might've caused a problem. And also that *some* of Sniffer users might have experienced the problem as well. It sounds like things are still being worked out. I just wanted some kind of verification that they were aware of the problem, were working on it, that they were in some way sorry about what happened...you know - the usual stuff. And I know that you are not an official rep of Appriver or anything, but presently you're all we have in that role ;) Thanks Kevin Pete McNeil wrote: > Hello Kevin, > > Friday, May 18, 2007, 8:52:47 PM, you wrote: > > >> Pete - Thanks for the reply, but I guess I don't understand what >> you're saying. "Some packet loss" and "rulebase downloads to slow >> down for a time" don't reflect what happened to me yesterday and >> apparently not what happened to one of the other posters either when >> he said that Appriver was having a problem "with sending messages >> over and over again". I received over (at last count) 35,000 >> messages (almost all of which were bounced replies, from one email >> from one of our users who sent an email to about 70 people) yesterday. >> > > >> And I had already gone to http://www.armresearch.com/ yesterday and >> there was nothing there. There is nothing there today that I can see. >> > > >> What happened? I lost an entire day's worth of email because of >> bounced messages. I didn't sleep last night. I don't even use >> Appriver. I would hope someone could explain it a little better than that. Thanks. >> > > I was answering the question - how is AppRiver related to Message > Sniffer. > > I don't have specifics on the problem at AppRiver yet - they are still > picking up the pieces, though operations are back to normal afaik. I > do know (preliminarily) that the problem occurred when a new piece of > software caused some messages with multiple recipients to loop and as > a result to be replicated and resent repeatedly. > > If you are not a user of AppRiver then you shouldn't have been > effected. Perhaps if you sent a message to someone who is a user of > AppRiver then that might have gotten your messages involved. > > The only direct effect I'm aware of for SNF users was that for a time > rulebase downloads were slowed due to packet loss. > > Since we use AppRiver for filtering (they, after all are using SNF) > some messages that get sent to us apparently did loop to some lists. > Also, some email to our accounts was delayed. > > I would need to know a lot more about your system and the email you > lost before I could make any guesses as to what happened there -- but > if you're not using AppRiver then you shouldn't have been effected. > > Hope this helps, > > _M > > # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to &l
[sniffer] Re: Appriver issue
I have something that I would also like to clear up. When I indicated that AppRiver had removed it's contact page, it likely just wasn't operating at the time that I was attempting to access it. Considering their issues, it would not be a surprise to see other issues like this caused, but it seemed suspicious since their home page was working and not their contact page. I did note that it was working by the time that it was pointed out that it was up. In no way did I ever believe that Pete or Sniffer had any direct involvement in the system that created these problems, and in no way should this reflect badly on Pete or Sniffer as far as I am concerned. I was slightly miffed after getting off the phone with them where their reaction quite clearly indicated that they were aware of the issue. I suggested that they take their servers off-line due to the issues that were being caused, but I was probably barking up the wrong tree. The servers weren't taken off line for another hour or so, or maybe this is when the delivery servers caught up with the queued E-mail destined for my client. I'm not sure why they didn't act on this sooner. When you have a loop, it is important to stop it, and their multi-homing made it difficult for others to block. One user received about 500 copies of the same message (and also called them), and there were other examples that we saw which were much more limited. I do hope that they didn't choose to introduce new software at 11 a.m. ET on the busiest E-mail day of the week, and that this was only when the problems surfaced... Everyone that deals with significant volumes of E-mail has issues from time to time, and I wouldn't draw conclusions about AppRiver based on just this one circumstance. I would imagine that it is hard to plan for how to deal with a broad scale looping issue, and I'm sure this was a learning experience for them. Matt David Moore wrote: I think what Peter is try to say is that Sort monster is hosted at Appriver and Appriver had an issue and therefore so did Sort monster. http://www.dnsstuff.com/tools/dnsreport.ch?&domain=sortmonster.com Regards David Moore [EMAIL PROTECTED] J.P. MCP, MCSE, MCSE + INTERNET, CNE. www.adsldirect.com.au for ADSL and Internet www.romtech.com.au for PC sales Office Phone: (+612) 9453 1990 Fax Phone: (+612) 9453 1880 Mobile Phone: +614 18 282 648 POSTAL ADDRESS: PO BOX 190 BELROSE NSW 2085 AUSTRALIA. - This email message is only intended for the addressee(s) and contains information that may be confidential, legally privileged and/or copyright. If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email, or taking any action in reliance on its contents by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Rogers Sent: Saturday, 19 May 2007 11:59 AM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue Thanks for the explanation, and I wasn't trying to blame you - just wanted more info is all. We use Sniffer, but not Appriver. You said that if we don't use Appriver, we shouldn't have been affected, but you also seemed to say that if one of the recipient's of my user's email uses Appriver that might've caused a problem. And also that *some* of Sniffer users might have experienced the problem as well. It sounds like things are still being worked out. I just wanted some kind of verification that they were aware of the problem, were working on it, that they were in some way sorry about what happened...you know - the usual stuff. And I know that you are not an official rep of Appriver or anything, but presently you're all we have in that role ;) Thanks Kevin Pete McNeil wrote: Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. "Some packet loss" and "rulebase downloads to slow down for a time" don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem "with sending messages over and over again". I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There
[sniffer] Re: Appriver issue
Inserting my 2 cents here since that is all that it is worth. In backing up what Matt said, let me relate a similar example of a problem that occurred a year and a half ago to a major IT security products vendor: At about 6:15 AM PT on a week day in the middle of a normal busy week, their content filtering servers begin to become unresponsive. At first, it was intermittent and hard to pinpoint. But within about 45 minutes, they stopped responding completely. Well, their appliances did what they were designed to do by default configuration, fail safe. Block all access if the content filtering server does not respond. All one had to do though was to log onto the appliance and change the failsafe block to allow. But this is where the fun (not) began. There are hundreds or more of library's, both public and private, as well as schools, that are using those appliances and that content filtering service. Guess what? They are bound by law to have content filtering in place, meaning they could not turn the fail safe off. Companies and schools and libraries started screaming bloody murder and demanded a resolution an hour ago. The content filtering service was finally restored about 2:30 PM if I recall correctly. So, what happened? I mean this is a big company and it should have things in place to prevent this. Right? They did. As much as some one would expect them to. They had 4 servers. The servers were fine, they were still running. There were no software changes, and in fact their tests showed the servers were still responding. They were located at a location with multiple internet connections, and all tests showed the internet connections were all up and working. Power was flowing fine and all UPSs as well as the generator were all fine. Finally, after about 2 hours, the problem was found: My understanding is that a single module in a enterprise router failed but in a way that was hard to find. Once found, the hardware vendor sent a replacement part by courier to replace. My understanding is that it cost them well over 10 grand to eliminate that one single point of failure. And that was just for the hardware. Just goes to prove once again that in IT, 80% of the result is 20% of the cost. That remain 20% of result is what costs the 80%. John T From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Friday, May 18, 2007 9:44 PM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue I have something that I would also like to clear up. When I indicated that AppRiver had removed it's contact page, it likely just wasn't operating at the time that I was attempting to access it. Considering their issues, it would not be a surprise to see other issues like this caused, but it seemed suspicious since their home page was working and not their contact page. I did note that it was working by the time that it was pointed out that it was up. In no way did I ever believe that Pete or Sniffer had any direct involvement in the system that created these problems, and in no way should this reflect badly on Pete or Sniffer as far as I am concerned. I was slightly miffed after getting off the phone with them where their reaction quite clearly indicated that they were aware of the issue. I suggested that they take their servers off-line due to the issues that were being caused, but I was probably barking up the wrong tree. The servers weren't taken off line for another hour or so, or maybe this is when the delivery servers caught up with the queued E-mail destined for my client. I'm not sure why they didn't act on this sooner. When you have a loop, it is important to stop it, and their multi-homing made it difficult for others to block. One user received about 500 copies of the same message (and also called them), and there were other examples that we saw which were much more limited. I do hope that they didn't choose to introduce new software at 11 a.m. ET on the busiest E-mail day of the week, and that this was only when the problems surfaced... Everyone that deals with significant volumes of E-mail has issues from time to time, and I wouldn't draw conclusions about AppRiver based on just this one circumstance. I would imagine that it is hard to plan for how to deal with a broad scale looping issue, and I'm sure this was a learning experience for them. Matt
[sniffer] Re: Appriver issue
My personal opinion is worth way less than John's, but I'd still like to insert it here. I was dramatically affected by a software product that I don't even subscribe to, so I'm somewhat curious why you would defend them so readily at this juncture. Perhaps they aren't totally to blame. But perhaps you are unaware of some of the ramifications of this foul-up. I'm not sure. But if you were affected by a service that you didn't have any connection to the way I was, perhaps it would be a different story. It seems like every message that was sent to appriver to be pattern-checked for potential spam, was then sent out to the intended recipient, *every time it was checked against their spam filters*. Which caused 1000 messages to be delivered per message, and then some, which caused a crazy amount of return or bounced messages in turn (which is where my server was hit). Again, I don't have all the facts and I may be wrong about some of the details, but this is what appears to have happened to me. Here is a snippet of one of the messages that was bounced back to my server, just FYI. This is just a snippet, and the headers were much longer, but I just wanted to throw them out there just in case. Received: from server128.appriver.com (HELO inbound.appriver.com) ([207.97.226.126]) by rrcs-mgw-01b.hrndva.rr.com with ESMTP; 17 May 2007 14:38:51 -0400 Received: by inbound.appriver.com (CommuniGate Pro PIPE 5.1.7) with PIPE id 7902890; Thu, 17 May 2007 14:19:43 -0400 Received: from [207.97.230.16] (HELO server97.appriver.com) by inbound.appriver.com (CommuniGate Pro SMTP 5.1.7) with ESMTP id 7902398; Thu, 17 May 2007 14:18:45 -0400 Received: by server97.appriver.com (CommuniGate Pro PIPE 5.1.4) with PIPE id 337776981; Thu, 17 May 2007 14:16:18 -0400 Received: from [74.205.4.33] (HELO inbound.appriver.com) John T (lists) wrote: Inserting my 2 cents here since that is all that it is worth. In backing up what Matt said, let me relate a similar example of a problem that occurred a year and a half ago to a major IT security products vendor: At about 6:15 AM PT on a week day in the middle of a normal busy week, their content filtering servers begin to become unresponsive. At first, it was intermittent and hard to pinpoint. But within about 45 minutes, they stopped responding completely. Well, their appliances did what they were designed to do by default configuration, fail safe. Block all access if the content filtering server does not respond. All one had to do though was to log onto the appliance and change the failsafe block to allow. But this is where the fun (not) began. There are hundreds or more of library's, both public and private, as well as schools, that are using those appliances and that content filtering service. Guess what? They are bound by law to have content filtering in place, meaning they could not turn the fail safe off. Companies and schools and libraries started screaming bloody murder and demanded a resolution an hour ago. The content filtering service was finally restored about 2:30 PM if I recall correctly. So, what happened? I mean this is a big company and it should have things in place to prevent this. Right? They did. As much as some one would expect them to. They had 4 servers. The servers were fine, they were still running. There were no software changes, and in fact their tests showed the servers were still responding. They were located at a location with multiple internet connections, and all tests showed the internet connections were all up and working. Power was flowing fine and all UPSs as well as the generator were all fine. Finally, after about 2 hours, the problem was found: My understanding is that a single module in a enterprise router failed but in a way that was hard to find. Once found, the hardware vendor sent a replacement part by courier to replace. My understanding is that it cost them well over 10 grand to eliminate that one single point of failure. And that was just for the hardware. Just goes to prove once again that in IT, 80% of the result is 20% of the cost. That remain 20% of result is what costs the 80%. *John T* *From:* Message Sniffer Community [mailto:[EMAIL PROTECTED] *On Behalf Of *Matt *Sent:* Friday, May 18, 2007 9:44 PM *To:* Message Sniffer Community *Subject:* [sniffer] Re: Appriver issue I have something that I would also like to clear up. When I indicated that AppRiver had removed it's contact page, it likely just wasn't operating at the time that I was attempting to access it. Considering their issues, it would not be a surprise to see other issues like this caused, but it seemed suspicious since their home page was working and not their contact page. I did note that it was working by the time that it was pointed out that it was up. In no way did I ever believe that Pete or
[sniffer] Re: Appriver issue
Hello Matt, Saturday, May 19, 2007, 12:44:25 AM, you wrote: > I was slightly miffed after getting off the phone with them where their reaction quite clearly indicated that they were aware of the issue. I suggested that they take their servers off-line due to the issues that were being caused, but I was probably barking up the wrong tree. The servers weren't taken off line for another hour or so, or maybe this is when the delivery servers caught up with the queued E-mail destined for my client. I'm not sure why they didn't act on this sooner. When you have a loop, it is important to stop it, and their multi-homing made it difficult for others to block. The response time was actually nearly immediate, but the effects lingered for a while due to a number of factors. I was one of the folks who detected the onset of this event when I detected packet loss. The entire combined technical team was engaged in the problem within minutes (single digits) of detecting the problem. The event presented as a DoS attack due to the heavy traffic and it's effects. After a short analysis (again, single digit minutes) we became aware of the true problem and the appropriate team began immediately correcting the issue. Shutting down the servers was not necessary and not a viable solution (though I'm sure it was considered). Even if that choice had been made the effects would have been the same due to the size of the system. That is - it would have taken the same amount of time to shut down the servers as it did to correct the software - and then there would have been significantly more collateral damage as a result. Given the circumstances the best choice was made and I'm amazed at how quickly the entire team was able to become positively involved (and coordinated) in solving the problem, mitigating damage, and recovering normal operations -- all while handling an understandably huge inrush of support calls. > Everyone that deals with significant volumes of E-mail has issues from time to time, and I wouldn't draw conclusions about AppRiver based on just this one circumstance. I would imagine that it is hard to plan for how to deal with a broad scale looping issue, and I'm sure this was a learning experience for them. Clearly, and thanks for that! There have already been a number of procedural changes and new tools developed from this event; the investigation is ongoing; and additional system changes will be forthcoming to help make these kinds of events far less likely, and even to help harden subsystems against the effects of these events whether they are caused unintentionally (such as this one) or otherwise. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Kevin, I don't quite understand. Do you, or do you not subscribe to appriver's hosted service? By the headers it appears you do. Thank You, Chris Bunting Lancaster Networks Direct: 717-278-6639 Office: 888-LANCNET x703 MS Certified Systems Engineer IP Telephony Expert Lancaster Networks 1085 Manheim Pike Lancaster PA 17601 www.lancasternetworks.com -- Corporate Technology Solutions... Specializing in 3com NBX Telephony Solutions IT Services - Phone Systems - Digital CCTV -- The information in this e-mail is confidential and may be privileged or subject to copyright. It is intended for the exclusive use of the addressee(s). If you are not an addressee, please do not read, copy, distribute or otherwise act upon this email. If you have received the email in error, please contact the sender immediately and delete the email. The unauthorized use of this email may result in liability for breach of confidentiality, privilege or copyright. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Rogers Sent: Saturday, May 19, 2007 7:37 AM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue My personal opinion is worth way less than John's, but I'd still like to insert it here. I was dramatically affected by a software product that I don't even subscribe to, so I'm somewhat curious why you would defend them so readily at this juncture. Perhaps they aren't totally to blame. But perhaps you are unaware of some of the ramifications of this foul-up. I'm not sure. But if you were affected by a service that you didn't have any connection to the way I was, perhaps it would be a different story. It seems like every message that was sent to appriver to be pattern-checked for potential spam, was then sent out to the intended recipient, *every time it was checked against their spam filters*. Which caused 1000 messages to be delivered per message, and then some, which caused a crazy amount of return or bounced messages in turn (which is where my server was hit). Again, I don't have all the facts and I may be wrong about some of the details, but this is what appears to have happened to me. Here is a snippet of one of the messages that was bounced back to my server, just FYI. This is just a snippet, and the headers were much longer, but I just wanted to throw them out there just in case. Received: from server128.appriver.com (HELO inbound.appriver.com) ([207.97.226.126]) by rrcs-mgw-01b.hrndva.rr.com with ESMTP; 17 May 2007 14:38:51 -0400 Received: by inbound.appriver.com (CommuniGate Pro PIPE 5.1.7) with PIPE id 7902890; Thu, 17 May 2007 14:19:43 -0400 Received: from [207.97.230.16] (HELO server97.appriver.com) by inbound.appriver.com (CommuniGate Pro SMTP 5.1.7) with ESMTP id 7902398; Thu, 17 May 2007 14:18:45 -0400 Received: by server97.appriver.com (CommuniGate Pro PIPE 5.1.4) with PIPE id 337776981; Thu, 17 May 2007 14:16:18 -0400 Received: from [74.205.4.33] (HELO inbound.appriver.com) John T (lists) wrote: > > Inserting my 2 cents here since that is all that it is worth. > > > > In backing up what Matt said, let me relate a similar example of a > problem that occurred a year and a half ago to a major IT security > products vendor: > > > > At about 6:15 AM PT on a week day in the middle of a normal busy week, > their content filtering servers begin to become unresponsive. At > first, it was intermittent and hard to pinpoint. But within about 45 > minutes, they stopped responding completely. Well, their appliances > did what they were designed to do by default configuration, fail safe. > Block all access if the content filtering server does not respond. All > one had to do though was to log onto the appliance and change the > failsafe block to allow. But this is where the fun (not) began. There > are hundreds or more of library's, both public and private, as well as > schools, that are using those appliances and that content filtering > service. Guess what? They are bound by law to have content filtering > in place, meaning they could not turn the fail safe off. Companies and > schools and libraries started screaming bloody murder and demanded a > resolution an hour ago. The content filtering service was finally > restored about 2:30 PM if I recall correctly. > > > > So, what happened? I mean this is a big company and it should have > things in place to prevent this. Right? > > > > They did. As much as some one would expect them to. > > > > They had 4 servers. The servers were fine, they were still running. > There were no sof
[sniffer] Re: Appriver issue
> My personal opinion is worth way less than John's, but I'd still like > to > insert it here. I was dramatically affected by a software product that > I don't even subscribe to, so I'm somewhat curious why you would defend > them so readily at this juncture. Perhaps they aren't totally to > blame. But perhaps you are unaware of some of the ramifications of > this > foul-up. I'm not sure. But if you were affected by a service that > you > didn't have any connection to the way I was, perhaps it would be a > different story. I am not so much defending the way the company handled it or such but am stating that hey things happen, lets not over react. And I understand completely. In my example, I nor my company nor my servers were using the content filtering that was involved. But just the night before while investigating a problem at the office of my biggest client, I found that there was a group of users accessing websites from the office that were causing problems bandwidth problems that those office managers had complained about. So I enabled the content filtering for all offices. I then sent an email to the management of the action I took pending further investigation. Well, at 7:00 AM the next morning, before I knew exactly what was happening or the extent of it, I had the CEO of the company on the phone screaming at me threatening legal action since their offices could not get on-line to process financial transactions that their customers were depending on. John T # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Appriver issue
Hello John, Saturday, May 19, 2007, 1:33:37 PM, you wrote: > So I enabled the content filtering for all offices. I then > sent an email to the management of the action I took pending further > investigation. Well, at 7:00 AM the next morning, before I knew exactly what > was happening or the extent of it, I had the CEO of the company on the phone > screaming at me threatening legal action since their offices could not get > on-line to process financial transactions that their customers were > depending on. This kind of thing (in general) is why keyboards without backspace keys never caught on. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
