On Tuesday, November 30, 2004, 10:15:18 PM, Jonathan wrote: J> I'm not quite sure I understand why you're going this direction. Obviously J> you have your reasons, and ultimately that's your business .. but .. :)
J> Isn't it a whole lot more efficient for things to be forwarded directly to J> you the moment our systems receive them? It seems like if your systems have J> to probe a pop3 box ever couple of minutes or more, it just takes that much J> longer to replicate through the system, and that much longer to come back J> into our rulebase. The only logic I can come up with, is maybe your "pop J> bot" will stop DL'ing the message, if it's already in the rulebase, thus J> saving a little bandwidth? Even so, it seems like that would far be J> outweighed by the bandwidth it takes to pop the boxes frequently. I'm happy to explain. As we expand we are dividing our operations across a divers set of networks and points of presence. Also, the sheer volume of data that we are handling is growing significantly. If we have all spamtraps directed to a given domain, then the easiest way to do that is for one central point to receive all of the incoming data. This means that there must be some centralized processor that must be large enough to manage all of that data, and that location must be public. By moving to a pop3 strategy, we can have our robots collect the data from diverse, anonymous locations anywhere on the 'net. These nodes can then be somewhat protected because they don't have to accept any connections that aren't from our own trusted networks, and they will always have complete control over their load since they are initiating and controlling the transactions. This makes it more difficult for these systems to be attacked and so critical components of our infrastructure will be able to operate without too much interference. It is also much easier to expand, move, modify, and decommission these processors as needed since they only answer to us. On the contrary - if we give out addresses where spam should be sent then that means there must be a server there to receive the messages - and that makes it a target. It also complicates the maintenance process because something must always be there to answer incoming connections. --- I suspect this explanation will raise some other questions so I will let out a bit more info about our future direction - with the disclaimer that I am not specifying any time-line :-) -- I want to avoid setting any unrealistic expectations. Here goes --- The production schedule has been adjusted and so a number of features have been lowered in their priority so that we can focus on moving to the Version 3 rulebase format. (We actually have plans through version 5 for Message Sniffer, we just don't talk about them much...) The version 3 rulebase format significantly reduces the processing costs for re-folding the token matrix, reduces the size of the matrix significantly (maybe as much as 70%), and uses a structure that is more readily modified incrementally. Once we have everyone moved to version 3x using this new rulebase format we will begin working on automated real-time update mechanisms. When these features are in place, SNF nodes will generally not download complete rulebases but will instead receive SMTP messages carrying encrypted XML packets that include incremental updates. These updates will be generated in real time as the rulebase is adjusted, and eventually as rules are generated by trusted peer nodes in the network. The effect of this will be to virtually eliminate the delays that are imposed by update scheduling along with most of the bandwidth requirements of the current model. If everything goes as planned, it will be common for new rules to be in place on client systems within a few hundred seconds of their creation. Here again, the update packets will be sent from widely distributed and potentially random anonymous systems throughout the 'net. Since these systems will only be transmitting updates, and since these updates will be transmitted in-band (SMTP), they will be similarly protected from most abuse. These first two steps insulate the critical operations of detecting new spam and distributing new rulebase updates while simplifying the task of expanding this system to serve our growing customer base. Eventually all of our critical operations will be carried out on a widely distributed, largely anonymous network of systems that only respond to our known trusted networks... to other networks they will simply be invisible (incoming packets from un-trusted networks will generally be nul routed). So, in summary, the move to pulling spamtraps via POP3 is part of a larger strategy to secure our critical infrastructure and ensure that we remain operational for our customers. There are some smaller, but still helpful reasons for moving in this direction also. For example, it is generally easier to set up a local mailbox than it is to set up scripting and forwarding mechanisms... so our customers should have an easier time dealing with this. As for time delays --- whether the robots pull the incoming messages out of our local server or directly off of our customer's servers, that polling process takes the same amount of time and has the same delay. In the grand scheme of things, we're actually saving a delay by going directly to the source --- so the message makes one fewer hop before it is seen by our systems. Another advantage of pulling the messages from our customers' servers is that the Received chain in the message is simplified. This helps us to see these messages in a "more pure" state. Of course there is a lot more to all of this, but I think I've covered the high points. Let me know if you have more questions. Thanks! _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
