Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On dinsdag, sep 9, 2003, at 19:41 Europe/Amsterdam, Dean Anderson wrote: Let's first define our goal before declaring it impossible to reach. Well, I think the goal has been stated: Create an abuse-free email protocol. That goal is impossible. Thus, we have abusable protocols. Ok, not going to argue on that point (not necessarily agreeing, though). Perhaps you have a different goal in mind, but it doesn't sound like you accept the premise that it impossible to create an abuse-free protocol. My goal is to drastically lower unsullicited bulk mail. It's not the occasional message that bothers me, but the dozens each day that try to exploit human kind's lowest common denominator. I believe bulk and unsollicited can both be determined objectively, so there should be ample avenues to explore here. The NCSC's definition refers to ANY communication not authorized by the security model. Note that the term "Covert Channel" is frequently associated with Multilevel Secure Operating Systems. The liturature uses other terms to describe the same concept: "information leakage", "sneaky signalling", "hidden data flows", "side channels". So you must not get too caught up in the peculiarities of operating systems. The concept is quite general. It is immediately obvious that covert channels have 3 characteristics: It is a CHANNEL, it is COVERT, and it VIOLATES SECURITY POLICY. (Caps for emphasis of definition, not loudness.) Ok. CHANNEL: Spam is a type of Email. Email is a channel transfering signals from A to B. Ok. COVERT: Spam is hidden in so far as possible from the system operators. It is an unintended communication in that the system operators intended that only non-broadcast, solicited commercial communication flow. This not an exclusive list of what is permitted, but illustrates that spam isn't permitted. Hm, is it? In most cases spamminess is pretty obvious, but I've both received and sent out complaints about spam which turned out to be fairly legitimate communication that just wasn't all that welcome any more. (But hard to tell when businesses change their (domain) name.) This stuff is hard to determine on a per-message basis. The part that I care about is that spam is unwelcome in two ways: each individual spam message is unwanted by the recipient, and the generated system load is unwanted by the service provider. VIOLATES SECURITY POLICY: System Operators specified an acceptable use policy that outlines what is permitted and what is not permitted. UCE is not permitted. Various methods have been implemented to enforce this policy. Again, too hard to determine by just looking at a message somewhere in the middle of the path. Now if we define that a single user may only send 1000 messages a day, that's something we can work with. If you can detect abuse on input rather than on output, detecting abuse is exactly what enables you to make an abuse-free protocol. Input to the first mailserver is output to the mail client. There may be many input/output connections. For every input, there is an equal and opposite output. (Seems that perpetual motion may have some analog after all ;-) Yes, this makes the whole thing somewhat harder... We are already "only allowing known bulk emailers". Untrue. Speak for yourself. You are the one who said "we". :-) I do think Shelby was on the right track with the idea of separating mailinglists from regular one-to-few email. (Not quite sold on the pop thing, though, that would be going 8 years back in time for me.) Mailinglists usually already have reasonable access controls and a single or very limited number of sources, so those shouldn't be a huge problem. With the sollicited bulk email out of the way, any bulk email that's left is automatically unsollicited (or a corner case that we can ignore at the price of some inconvenience) so this can be stopped or slowed down. Spammers can _always_ do whatever regular users can do. There is no way to authenticate regular users and not authenticate spammers. This is why SMTP AUTH has no effect on spam. If I know who sent something I can filter so I don't have to receive any future filth from the same source. And if spammers are forced to use their real identity then finally they'll get at least some of what they have coming. If regular users can inject arbitrary addresses, then so can spammers. Regular users can always inject arbitrary addresses. Ordinarilly, this is desired. Disagree. The point being? The point being that the persons sending most of the spam will do anything to annoy people. It has no grounding in commercial motivation, nor can regulation of commercial activity have any effect. In other words, protocols that require cooperation of the spammer aren't likely to succeed. What we need is a system where the only action that has any result is a valid action. Then either spammers have to spew out their filth out in the open without being able to hide, and bear the
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Very nice. I say to post an Internet Draft - you post a link to a simple
archived e-mail. The IETF process starts with an Internet Draft - without it
we are all just wasting time. An internet draft is a concrete proposal that
can be discussed, archived, debated successfully, etc.
I challenge you to take your e-mail posting and turn it into an Internet Draft
with a legitimate security section (since you are solving a security problem)
then post a single message to this list showing the internet draft, and a
mailing list that people can join if they are interested in discussing it
further.
>From there it is easy to determine if there is enough interest in forming a BOF
in Minnesota ( or S. Korea in March ) to forward the work in a Working Group.
The problem you have run into is you haven't taken the first step, which is to
simply submit an Internet Draft to the Internet Drafts editor... very simple
process with no politics in the way. From there you can pursue forming a
Working Group (where you will get your first taste of politics). Being that
you haven't taken the first step (publishing an Internet Draft) I am not sure
you "really" meet the charter (Ok, yes you do, but putting out a draft is SO
important - it should be the first step) and you have allowed the topic to grow
WAY beyond initial discussions (I am actually waiting for Harold to post one
of his famous Top n Talker lists soon). The next step is to get a mailing
list somewhere and move the discussion off of this list onto that list
Bill
On Wed, Sep 10, 2003 at 05:10:06AM +0800, Shelby Moore wrote:
> >Why is this even difficult. I have yet to see a firm proposal (ie. an
> >Internet Draft),...
> >My challenge - Go forth - publish your protocol in ID form...
>
>
> 1. I remind you to read my initial post that started this thread:
>
> http://www1.ietf.org/mail-archive/ietf/Current/msg22035.html
>
> "Request for opinions on whether to creating a working group or publish the
> following idea as an internet draft?"
>
> I think that qualifes under "initial discussion" of the charter of this list (see #2
> below).
>
>
> 2. And to read the charter for this mailing list:
>
> http://www.ietf.org/maillist.html
>
> "This list is meant for initial discussion only. Discussions that fall within the
> area of any working group or well established list should be moved to such more
> specific forum..."
>
>
> 3. Just a fews posts back in this thread, it was suggested that IRTF would be a
> better forum for anti-spam proposals and discussions, and I agreed (to consider it
> if possible and applicable).
>
> However there is a another line of discussion in this thread pertaining to general
> information theory and modeling of channels which is still winding down ("initial
> discussion") and seems quite general to internet engineering.
>
>
> 4. The basic difficulty has been those violating the charter of the list:
>
> http://www.ietf.org/maillist.html
>
> "Unprofessional commentary, regardless of the general subject." such as the "Kook"
> thread offshoot that someone started.
>
> Shelby Moore
> http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Wed, 10 Sep 2003, Shelby Moore wrote: > At 01:41 PM 9/9/2003 -0400, you wrote: > > However, I think the analysis of the concepts of information theory, > channels, and models of spam is more fundamental to "internet > engineering" than the original purpose of this thread and thus I see no > reason why it would not be useful data here at IETF. I tend to think that the more theoretical aspects of abuse-free protocols aren't specific to spam, and are probably of more general interest. However, discussion needs to be productive, as well as interesting. > Before I respond to your continuance of your argument, I *respectfully* > remind that I already refuted the whole line of criticism you are > continuing in this post, when I rebutted your last post in this thread: I did read your post. But it seems we have unbridgeable differences: > but to focus your attention on detection, rather than protocol > alteration. It is impossible to alter the protocol in any way that > will force the spammer to identify themselves a-priori as a spammer. Disagree strongly. First benefit is once you define spam == *BE (instead of UBE), then it is easier to model spam and do research on it,because you can model it at any node in the channel, not only at the receiver end point. That was my whole point about "enforcers". I think I've already covered this, but perhaps more clarity will be helpful: The theoretical definition of "spam" is simply "email that is unintended by the the system operator, that is hidden from the system operator, and violates the security policy of the system operator". No more precision is necessary to use the theoretical framework I am using. We don't need the exact intentions of the system operator, nor the security policy. We just have to posit that these exist, and that they are violated by spam. For example, altering the definition of spam doen't have any effect on the impossibility of making a protocol that will automatically force the spammer (uncooperatively) to distingish themselves from regular users (self marking). Self-marking could only be done with the spammers cooperation, and there is no reason to think that todays abusers will cooperate, since most of those abusers are already unconvicted felons and have no commercial purposes. Frequently, these abusers appear to use viruses, which could be using the credentials of regular users. So, they can do _everything_ those regular users can do. Because you can't count on any quick cooperation before the recipients mailbox, it doesn't seem to be helpful to think about modeling spam at other nodes. --Dean
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I started this thread. Let's please close this thread here on IETF and move it some where more appropriate. I do not know exactly where to move it to (which list I will choose exactly because I want to research more to try to avoid running into Vernon, Valdis, Spencer, and others again which will be difficult since they seem to be every where). What I may do is write the LD and make a working group some where and then apply to such a list and ask only those are really interested to follow. Sometime in the future, I will make one more post on this list to say where all my replies will be going. So if there are any more posts, I will be collecting them and making replies on them to another list in future. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Note that advertising is inappropriate on the IETF mailing list. I don't think that was Vernon's or my intention. Then remove both Vernon's (DCC) and my post. I saw it as implemented examples of two different fundamental internet engineering ways of dealing with out of signal email. Both the DCC and ours are free services as best as I can tell, so don't know what the advertising would be for. The traffic here is miniscule compared to a link I can put on other sites I run and I would guess probably similarly for Vernon. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Note that advertising is inappropriate on the IETF mailing list. --On 10. september 2003 05:21 +0800 Shelby Moore <[EMAIL PROTECTED]> wrote: If we comparing solutions to viral attachments (that also do anti-spam), ours operates after the MTA but before (statistically) the MUA, and is instantly user configurable:
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> >Oh, please tell me you're not going to keep posting pointers to your > >previous postings until everyone agrees with you. > > If Dave Anderson (or any one else) keeps making new posts *misstating* that my proposal is to make an abuse-free protocol, when my proposal is not, then I guess I might have to keep clarifying that my proposal is not to make an abuse-free protocol, eh? Apparently Randy Bush is unavailable, so I will repeat his previous post ... > troll? what troll? procmail is your friend. Spencer (with apologies to Randy, but I bet he isn't too upset) http://www1.ietf.org/mail-archive/ietf/Current/msg22117.html
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Oh, please tell me you're not going to keep posting pointers to your >previous postings until everyone agrees with you. If Dave Anderson (or any one else) keeps making new posts *misstating* that my proposal is to make an abuse-free protocol, when my proposal is not, then I guess I might have to keep clarifying that my proposal is not to make an abuse-free protocol, eh? I would help to actually read the post before replying, in order to cut down the noise level. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Reports from some operators of DCC clients at non-trivial sites >claim that the DCC does a tolerable job against SoBig.F. >... I'd not expect the DCC to do >well against most worms or viruses. I agree in that it seems to me on an internet engineering level of analysis, it makes a lot more sense to remove the viral attachments BEFORE the MUA. Afaik, the DCC (usually) works at the MTA layer which is nice for system admins but perhaps not ideal for users that don't have system admins. If we comparing solutions to viral attachments (that also do anti-spam), ours operates after the MTA but before (statistically) the MUA, and is instantly user configurable: http://AntiViotic.com/test.php Will solve 100% your viral attachment problem with the following advantages: http://AntiViotic.com/antiviotic.php Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Why is this even difficult. I have yet to see a firm proposal (ie. an
>Internet Draft),...
>My challenge - Go forth - publish your protocol in ID form...
1. I remind you to read my initial post that started this thread:
http://www1.ietf.org/mail-archive/ietf/Current/msg22035.html
"Request for opinions on whether to creating a working group or publish the following
idea as an internet draft?"
I think that qualifes under "initial discussion" of the charter of this list (see #2
below).
2. And to read the charter for this mailing list:
http://www.ietf.org/maillist.html
"This list is meant for initial discussion only. Discussions that fall within the area
of any working group or well established list should be moved to such more specific
forum..."
3. Just a fews posts back in this thread, it was suggested that IRTF would be a better
forum for anti-spam proposals and discussions, and I agreed (to consider it if
possible and applicable).
However there is a another line of discussion in this thread pertaining to general
information theory and modeling of channels which is still winding down ("initial
discussion") and seems quite general to internet engineering.
4. The basic difficulty has been those violating the charter of the list:
http://www.ietf.org/maillist.html
"Unprofessional commentary, regardless of the general subject." such as the "Kook"
thread offshoot that someone started.
Shelby Moore
http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Oh, please tell me you're not going to keep posting pointers to your previous postings until everyone agrees with you. Spencer - Original Message - From: "Shelby Moore" <[EMAIL PROTECTED]> [deleted down to ] > > Before I respond to your continuance of your argument, I *respectfully* remind that I already refuted the whole line of criticism you are continuing in this post, when I rebutted your last post in this thread: > > http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html > > In case any one missed it, [deleted down to] > > > No that is not the stated goal of this thread I started. I already rebutted that whole link of criticism here: > > http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html > > Look for the section that starts with: > "Your point is that it is futile to define a protocol..." > > > And here: > > http://www1.ietf.org/mail-archive/ietf/Current/msg22129.html > [deleted down to] > > > The links to the previous posts are above which state that is not our goal. You have been told that at least 2 or 3 times already. > [deleted down to ] "pull"rather than repeat my entire logic here, please read the linked posts above in entirety. > [deleted down to] > > And COVERT has nothing to do with my proposal as I've detailed ad nauseum in the above linked posts. > [deleted down to ] > > Again read the linked posts above more carefully. With a different model of spam, we aren't stopping abuse, we are merely increasing detection by having a better model of the signal. > > [deleted down to] > > > This thread is not proposing that. See above. > >
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>grenville armitage wrote: >Valdis gave you the initial pointer, why should he do your literature review >(prior art?) research as well? grenville armitage, Afair, I think you've sent this post something like 6 times already, which I've tried to ignore. It seems noise (spam) the only logical rebuttal you have to my proposal against spam. Furthermore, I have searched and I have not found any one suggesting exactly what I have proposed here. I have challenged Valdis to provide a link to prior art which proposes what I have proposed and Valdis has not done so (yet). Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:41 PM 9/9/2003 -0400, you wrote: >My apologies for this message. This discussion is winding down. Iljitsch >makes some interesting points, to which I have tried to respond >thoughtfully. Dean, Yes as already stated, I do intend to close this thread and eventually provide a forwarding link to a new discussion elsewhere (perhaps at IRTF as someone suggested, if possible...) However, I think the analysis of the concepts of information theory, channels, and models of spam is more fundamental to "internet engineering" than the original purpose of this thread and thus I see no reason why it would not be useful data here at IETF. Before I respond to your continuance of your argument, I *respectfully* remind that I already refuted the whole line of criticism you are continuing in this post, when I rebutted your last post in this thread: http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html In case any one missed it, I think the most relevant section there begins done the page a bit with: "Disagree strongly. First benefit is once you define spam == *BE (instead of UBE), then it is easier to model spam..." more below... >> Your analogy doesn't fly. Our email protocols have holes big enough to >> drive a truck through. Is it unreasonable when people ask the IETF >> leadership for a place to work on this? > >I don't think our email protocols have any holes at all. They can be >abused. But mere abuse is not a "hole". Semantics debate only. Better to stick to the real point below... >> > "We", meaning the IETF, care, because this is very useful aid to >> > deciding what to work on. We know that we need to focus on leak >> > stoppage, not trying to invent leak-proof protocols. There is no >> > point researching something that is impossible. >> >> Let's first define our goal before declaring it impossible to reach. > >Well, I think the goal has been stated: Create an abuse-free email >protocol. No that is not the stated goal of this thread I started. I already rebutted that whole link of criticism here: http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html Look for the section that starts with: "Your point is that it is futile to define a protocol..." And here: http://www1.ietf.org/mail-archive/ietf/Current/msg22129.html Start reading down from: "I proposed an way to improve leak stoppage, by defining the signal in the channel and not only at end points. I never proposed a leak-proof protocol." >Perhaps you have a different goal in mind, but it doesn't sound like you >accept the premise that it impossible to create an abuse-free protocol. The links to the previous posts are above which state that is not our goal. You have been told that at least 2 or 3 times already. >> Iljitsch van Beijnum wrote: >> The jump from "spam" to "covert channel" isn't immediately obvious. And >> if any proof of why spam is a covert channel has been offered, I've >> managed to miss it. Iljitsch van Beijnum, I think what Dean Anderson means is that because you can't create a 100% perfect covert channel, then spammers will always find a way to abuse, no matter what you do on the protocol level. Theoretically I agree with him. However, he is ignoring the posts I made (as linked above), which show that is not what I am proposing. What I am proposing has to do with improving the model of spam so it can be more easily detected at more points in the channels and earlier and other detection advantages. To get this model, I propose that we need a new definition of legitimate bulk email, from "push" to "pull"rather than repeat my entire logic here, please read the linked posts above in entirety. > Dean Anderson wrote: >The NCSC's definition refers to ANY communication not authorized by the >security model. Note that the term "Covert Channel" is frequently >associated with Multilevel Secure Operating Systems. The liturature uses >other terms to describe the same concept: "information leakage", "sneaky >signalling", "hidden data flows", "side channels". So you must not get too >caught up in the peculiarities of operating systems. The concept is quite >general. And COVERT has nothing to do with my proposal as I've detailed ad nauseum in the above linked posts. >CHANNEL: Spam is a type of Email. Email is a channel transfering signals >from A to B. Email is really a subchannel of the internet, which is a >subchannel of the PSTN, public and private fiber networks, etc. And moving legitimate bulk email to a "pull" channel is part of my proposal. >COVERT: Spam is hidden in so far as possible from the system operators. It >is an unintended communication in that the system operators intended that >only non-broadcast, solicited commercial communication flow. This not an >exclusive list of what is permitted, but illustrates that spam isn't >permitted. Part of my overall point made in the links of posts above is that one of the reasons it is "hidden" is because it can on
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Why is this even difficult. I have yet to see a firm proposal (ie. an Internet Draft), and once there is one, it is a simple matter of asking an AD to sponsor a BOF to see if there is interest in forming a working group to solve the problem. I remember sitting through several YATP (Yet another Tunnelling Protocol) BOFs years ago, I don't see what the problem is with creating some YASPP BOFs (Yet Another Spam Prevention Protocol). This is the IETF, that is the IETF process... Why are we arguing here about killing it without having a firm proposal, and a BOF chartered to form a WG to go solve the problem. Any more breath we waste now doesn't help anybody. My challenge - Go forth - publish your protocol in ID form, contact the correct APP (probably) area AD and get a BOF in Minneapolis - Convince the IETF in general there that a WG should be chartered to solve the problem. Go and solve the problem Bill On Tue, Sep 09, 2003 at 01:41:33PM -0400, Dean Anderson wrote: > My apologies for this message. This discussion is winding down. Iljitsch > makes some interesting points, to which I have tried to respond > thoughtfully. > > --Dean > > On Tue, 9 Sep 2003, Iljitsch van Beijnum wrote: > > > On maandag, sep 8, 2003, at 17:30 Europe/Amsterdam, Dean Anderson wrote: > > > > >> Nobody cares. Making a roof 100.00% impervious to water molecules > > >> may be impossible, but that doesn't mean we have to resign to getting > > >> wet every time it rains. > > > > > People care because when someone comes around saying "you can have a > > > 100% > > > impervious roof if only you jump through these inconvenient hoops", we > > > know that they are wrong, and don't need to waste time considering how > > > inconvenient the hoops are. > > > > Your analogy doesn't fly. Our email protocols have holes big enough to > > drive a truck through. Is it unreasonable when people ask the IETF > > leadership for a place to work on this? > > I don't think our email protocols have any holes at all. They can be > abused. But mere abuse is not a "hole". > > > > "We", meaning the IETF, care, because this is very useful aid to > > > deciding what to work on. We know that we need to focus on leak > > > stoppage, not trying to invent leak-proof protocols. There is no > > > point researching something that is impossible. > > > > Let's first define our goal before declaring it impossible to reach. > > Well, I think the goal has been stated: Create an abuse-free email > protocol. That goal is impossible. Thus, we have abusable protocols. > > Perhaps you have a different goal in mind, but it doesn't sound like you > accept the premise that it impossible to create an abuse-free protocol. > > > >>> After I first posted this on IETF a while back, someone suggested > > >>> that covert channels require cooperation, and that spam therefore > > >>> isn't a covert channel. > > > > >> Where does this covert channel stuff come from anyway? > > > > > What do you mean? > > > > The jump from "spam" to "covert channel" isn't immediately obvious. And > > if any proof of why spam is a covert channel has been offered, I've > > managed to miss it. > > The NCSC's definition refers to ANY communication not authorized by the > security model. Note that the term "Covert Channel" is frequently > associated with Multilevel Secure Operating Systems. The liturature uses > other terms to describe the same concept: "information leakage", "sneaky > signalling", "hidden data flows", "side channels". So you must not get too > caught up in the peculiarities of operating systems. The concept is quite > general. > > It is immediately obvious that covert channels have 3 characteristics: It > is a CHANNEL, it is COVERT, and it VIOLATES SECURITY POLICY. (Caps for > emphasis of definition, not loudness.) > > > CHANNEL: Spam is a type of Email. Email is a channel transfering signals > from A to B. Email is really a subchannel of the internet, which is a > subchannel of the PSTN, public and private fiber networks, etc. > > COVERT: Spam is hidden in so far as possible from the system operators. It > is an unintended communication in that the system operators intended that > only non-broadcast, solicited commercial communication flow. This not an > exclusive list of what is permitted, but illustrates that spam isn't > permitted. > > VIOLATES SECURITY POLICY: System Operators specified an acceptable use > policy that outlines what is permitted and what is not permitted. UCE is > not permitted. Various methods have been implemented to enforce this > policy. > > > > >> the spammer's achilles heel is that they need to send out incredible > > >> amounts of email in order to fulfill their objectives, whichever > > >> those are. Detecting bulk mail is doable, and it shouldn't be too > > >> hard to come up with something to differentiate legitimate bulk > > >> emailing from spam. For instance, we can reverse the burden of proof > > >> here and only allow
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
My apologies for this message. This discussion is winding down. Iljitsch makes some interesting points, to which I have tried to respond thoughtfully. --Dean On Tue, 9 Sep 2003, Iljitsch van Beijnum wrote: > On maandag, sep 8, 2003, at 17:30 Europe/Amsterdam, Dean Anderson wrote: > > >> Nobody cares. Making a roof 100.00% impervious to water molecules > >> may be impossible, but that doesn't mean we have to resign to getting > >> wet every time it rains. > > > People care because when someone comes around saying "you can have a > > 100% > > impervious roof if only you jump through these inconvenient hoops", we > > know that they are wrong, and don't need to waste time considering how > > inconvenient the hoops are. > > Your analogy doesn't fly. Our email protocols have holes big enough to > drive a truck through. Is it unreasonable when people ask the IETF > leadership for a place to work on this? I don't think our email protocols have any holes at all. They can be abused. But mere abuse is not a "hole". > > "We", meaning the IETF, care, because this is very useful aid to > > deciding what to work on. We know that we need to focus on leak > > stoppage, not trying to invent leak-proof protocols. There is no > > point researching something that is impossible. > > Let's first define our goal before declaring it impossible to reach. Well, I think the goal has been stated: Create an abuse-free email protocol. That goal is impossible. Thus, we have abusable protocols. Perhaps you have a different goal in mind, but it doesn't sound like you accept the premise that it impossible to create an abuse-free protocol. > >>> After I first posted this on IETF a while back, someone suggested > >>> that covert channels require cooperation, and that spam therefore > >>> isn't a covert channel. > > >> Where does this covert channel stuff come from anyway? > > > What do you mean? > > The jump from "spam" to "covert channel" isn't immediately obvious. And > if any proof of why spam is a covert channel has been offered, I've > managed to miss it. The NCSC's definition refers to ANY communication not authorized by the security model. Note that the term "Covert Channel" is frequently associated with Multilevel Secure Operating Systems. The liturature uses other terms to describe the same concept: "information leakage", "sneaky signalling", "hidden data flows", "side channels". So you must not get too caught up in the peculiarities of operating systems. The concept is quite general. It is immediately obvious that covert channels have 3 characteristics: It is a CHANNEL, it is COVERT, and it VIOLATES SECURITY POLICY. (Caps for emphasis of definition, not loudness.) CHANNEL: Spam is a type of Email. Email is a channel transfering signals from A to B. Email is really a subchannel of the internet, which is a subchannel of the PSTN, public and private fiber networks, etc. COVERT: Spam is hidden in so far as possible from the system operators. It is an unintended communication in that the system operators intended that only non-broadcast, solicited commercial communication flow. This not an exclusive list of what is permitted, but illustrates that spam isn't permitted. VIOLATES SECURITY POLICY: System Operators specified an acceptable use policy that outlines what is permitted and what is not permitted. UCE is not permitted. Various methods have been implemented to enforce this policy. > >> the spammer's achilles heel is that they need to send out incredible > >> amounts of email in order to fulfill their objectives, whichever > >> those are. Detecting bulk mail is doable, and it shouldn't be too > >> hard to come up with something to differentiate legitimate bulk > >> emailing from spam. For instance, we can reverse the burden of proof > >> here and only allow know bulk emailers. > > > "Detecting abuse" is quite different from making a protocol that can't > > be abused. > > If you can detect abuse on input rather than on output, detecting abuse > is exactly what enables you to make an abuse-free protocol. And again: > just because there are situations that you can't do something doesn't > mean it's automatically useless to perform the action under other > circumstances too. Input to the first mailserver is output to the mail client. There may be many input/output connections. For every input, there is an equal and opposite output. (Seems that perpetual motion may have some analog after all ;-) > > But that is my point: You have to focus on detection. This > > doesn't require any protocol changes whatsover. > > Do you mean first accepting the message, then searching for anatomical, > pharmaceutical or financial terms and subsequently discarding the > message? This is useless as it only makes the spammers spam more, not > less. > > > We are already "only allowing known bulk emailers". > > Untrue. Speak for yourself. > > Unfortunately, that doesn't prevent spam. > > It should help if used to
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> > The viruses can use the credentials of the infected user. That is > > "legitimate", until someone reading the email realizes its not and > > complains. These send 40-50 messages per IP, and is hard to detect as > > bulk. Reports from some operators of DCC clients at non-trivial sites claim that the DCC does a tolerable job against SoBig.F. This is without the Greylist support now available in the DCC client code. The DCC detects bulk mail, defined as substantially identical messages from any SMTP client senders. I'd not expect the DCC to do well against most worms or viruses. SoBig is somewhat different. (I won't talk those differences in public or with people I don't know well enough to say they'll also be descrete. Like other people who care more about fighting viruses and spam than being known as fighters of viruses and spam, I think the profit in idle chatter is not worth the cost of giving even trivial aid and comfort to the bad guys.) As has been pointed out, all of this belongs in the ASRG mailing list if anywhere. See http://irtf.org/charters/asrg.html and https://www1.ietf.org/mail-archive/working-groups/asrg/current/maillist.html Vernon Schryver[EMAIL PROTECTED]
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On maandag, sep 8, 2003, at 17:30 Europe/Amsterdam, Dean Anderson wrote: Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. People care because when someone comes around saying "you can have a 100% impervious roof if only you jump through these inconvenient hoops", we know that they are wrong, and don't need to waste time considering how inconvenient the hoops are. Your analogy doesn't fly. Our email protocols have holes big enough to drive a truck through. Is it unreasonable when people ask the IETF leadership for a place to work on this? "We", meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. Let's first define our goal before declaring it impossible to reach. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? What do you mean? The jump from "spam" to "covert channel" isn't immediately obvious. And if any proof of why spam is a covert channel has been offered, I've managed to miss it. the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. "Detecting abuse" is quite different from making a protocol that can't be abused. If you can detect abuse on input rather than on output, detecting abuse is exactly what enables you to make an abuse-free protocol. And again: just because there are situations that you can't do something doesn't mean it's automatically useless to perform the action under other circumstances too. But that is my point: You have to focus on detection. This doesn't require any protocol changes whatsover. Do you mean first accepting the message, then searching for anatomical, pharmaceutical or financial terms and subsequently discarding the message? This is useless as it only makes the spammers spam more, not less. We are already "only allowing known bulk emailers". Untrue. Unfortunately, that doesn't prevent spam. It should help if used together with some kind of authentication (for which, I'm happy to see, there are several independently submitted proposals on the table) so that spammers can't inject arbitrary from addresses. Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The point being? The viruses can use the credentials of the infected user. That is "legitimate", until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. But when added up over a lot of IP addresses, is quite obviously annoying. Not as annoying as "real" spam. (Although the size of email worms is unpleasant. Fortunately, regular spam is usually pretty small.) Fixable with authentication. No, that's the point. It isn't _fixable_ with authentication. It isn't fixable at all. It is only "fixed" when the spammer loses his account. Then the spammer gets a new account. So it isn't really fixed. It is when they lose their accounts faster than they can acquire new ones. So we are always going to be playing a game of whack-a-mole. So? I don't mind doing that if I can win. Having to play and then be forced to lose is much harder to swallow. That cannot be avoided by altering the protocol or the authentication scheme (information theory proves this). So it is useful, then, to work on ways of detection, and improve our whack-a-mole skills. Altering protocols and authentication is a waste of time. I find it unacceptable that my email software lets people bombard me with crap while at the same time faking the headers such that said crap seems to come from an innocent bystander. I can't understand how any reasonable person can think this is ok. Or maybe we should just go ahead and move the "From: " header to historic and be done with it.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
After this issue, I am probably moving the thread to IRTF (as suggested) if possible
(but probably after taking a break to do some other work).
>> >> > Information theory says that such things are impossible. One can not
>> >> > construct a spam-free protocol because this is the same problem as
>> >> > constructing a system free of covert channels, which information theory
>> >> > says is impossible.
>>
>>
>> But information theory also says you can optimize signal-to-noise ratio,
>> but only if you know what the characteristics of your signal are.
>
>It actually doesn't say that precisely. It says that you can transmit a
>signal with an arbitrarilly low error rate at a speed below the channel
>capacity.
>
>The concrete task of altering the signal to noise ratio is accomplished by
>enhancing the signal with a harmonic oscillator, so that it is stronger
>than the noise.
Agreed.
And thus on a "conceptual level", you have to have some idea about the signal
characteristics in order to enhance it.
Actually if I remember correctly, your example is how it applies to periodic signals.
The general case is more abstract.
> This is then described as a set of differential equations
>that can be optimized with Variational methods. The limits of this
>process are indicated by information theory, the nyquist theorem, etc.
Add Shannon entropy, chaos, etc...
>If the channel isn't described by a fourier series, then the differential
>equations may not be solvable, and it may be impossible to optimize its
>signal to noise ratio. (Well, there are other mathematical methods, but
>you get the point.)
Yes that is what I meant that the general case is more abstract, so I was talking on a
"conceptual" or abstract level.
> You are borrowing the concepts by metaphor, but the
>concrete methods don't transfer well.
I was only using it to say we must define the signal how it appears in the channel
before we can do any research on it in the channel.
The way spam is currently defined defined as UBE (instead of my proposed *BE), then it
means you can only model the signal at the end point. Given that means in the
receivers subjective mind, that is not all that useful for research, unless you want
to get into very fuzzy science such pyschology. If you want to make the point about
practicality, then that is a very strong one!
>My point is not to discourage you from trying to stop spam,
You are only 1 of 3 people so far at IETF who has said that to me. The rest who have
commented have tried to discourage me. So thank you.
> but to focus
>your attention on detection, rather than protocol alteration. It is
>impossible to alter the protocol in any way that will force the spammer to
>identify themselves a-priori as a spammer.
Disagree strongly. First benefit is once you define spam == *BE (instead of UBE),
then it is easier to model spam and do research on it, because you can model it at any
node in the channel, not only at the receiver end point. That was my whole point
about "enforcers".
However, there is a problem. Some *BE is solicited. Which is why I proposed moving
the solicited *BE to another channel ("pull").
Your point is that it is futile to define a protocol that will separate the solicited
from the unsolicited, because spammers will always be able to subvert the protocol.
And you to say thus there are no benefits to detection. I strongly disagree. There
are two aspects to my response:
1. Spam coming thru the alternate "pull" channel can be modeled differently that spam
defined as *BE. This separation of models provides benefits over trying to model spam
as UBE in the receiver's mind (end point). Other person in this thread has provided
one specific example, which is the "pull" delay gives a whole new dynamic to
detection. Also I have pointed about that the membership quality of the solicited
channel, gives it unique modeling advantages.
2. Spam coming thru the existing channel can then be modeled as *BE at any node of the
channel, instead of as UBE. Some nodes have a much better model of spam in this
definition, than the one at the end point. For example, ISPs can see a lot more abuse
data in real-time, than a single receiver or the current inherently more clumsy
attempts to group or poll receivers.
Hopefully that will set the record straight that I am thinking about spam in new
conceptual ways...and not rehashing as others have claimed...
>You could ask for spammers to cooperatively self-mark their messages.
>But this hasn't been terribly productive.
Obviously I am not asking for that or any thing like that. See above.
> It is also pointless to ask for
>cooperative identification of non-spammers and identify spammers as those
>not in the set of non-spammers.
I am also not asking for this, and it is instructive to understand how I am not.
I am only making a definition, so that one can model under the benefits of that
definition. What people
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>I am not talking about email spreading virues. A number of viruses appear >to send spam. (not spreading). Sometimes this is autonymous. Sometime it >is under control via IRC channel back to the virus operator. > Further, it >seems that many open proxies are installed by virus. Once the virus has >control of the computer, it has or will obtain keys to private keychains, >etc. It can do whatever the infected users can do. > >The number of 40-50 emails per IP figure comes from analysis of spam >messages that get by filters, by reviewing how many messages came from the >same source. A lot of spam that gets by filters is of this very low volume >type. Dean this is important data for me, but not as useful until I can get you to tell me: 1. What was the time duration of the sample? 2. What was your total approximate sample size? 3. What was the total volume of this type of spam within that sample? Estimates would be fine. I would really appreciate it very much. Thanks, Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Tue, 9 Sep 2003, Shelby Moore wrote: > > >> > Information theory says that such things are impossible. One can not > >> > construct a spam-free protocol because this is the same problem as > >> > constructing a system free of covert channels, which information theory > >> > says is impossible. > > > But information theory also says you can optimize signal-to-noise ratio, > but only if you know what the characteristics of your signal are. It actually doesn't say that precisely. It says that you can transmit a signal with an arbitrarilly low error rate at a speed below the channel capacity. The concrete task of altering the signal to noise ratio is accomplished by enhancing the signal with a harmonic oscillator, so that it is stronger than the noise. This is then described as a set of differential equations that can be optimized with Variational methods. The limits of this process are indicated by information theory, the nyquist theorem, etc. If the channel isn't described by a fourier series, then the differential equations may not be solvable, and it may be impossible to optimize its signal to noise ratio. (Well, there are other mathematical methods, but you get the point.) You are borrowing the concepts by metaphor, but the concrete methods don't transfer well. My point is not to discourage you from trying to stop spam, but to focus your attention on detection, rather than protocol alteration. It is impossible to alter the protocol in any way that will force the spammer to identify themselves a-priori as a spammer. You could ask for spammers to cooperatively self-mark their messages. But this hasn't been terribly productive. It is also pointless to ask for cooperative identification of non-spammers and identify spammers as those not in the set of non-spammers. It may be too strong to say this is pointless, as qsecretary and things that do this are fairly popular, However, they don't quite solve the general case of things that send mail but don't receive mail. And of course such schemes are fooled by disposal accounts, which can respond until they are shut down. If everyone used qsecretary, spammers would simply alter their software to send responses. So given a set of unmarked messages, some spam, some not-spam, the task is to have a program mark them in the same way that a human would if a human were reading the messages. Since humans have different definitions of spam, it would be useful if the program could accept different definitions as well. This is the realm of content analysis. > Thus my whole motivation for an unambiguous definition (spam == all bulk > email) along the channel and not just a definition at the end points > (UBE). You may need a precise definition before you can begin implementation (just like you need a definition of voltage, current, etc to begin building a transmitter), but you do not need a precise definition to talk about the theoretical aspects. Spam could be defined as UCE, CE, UBE, or BE. I have also a more complete and detailed taxonomy of spam: There are 3 types of email that we generally call spam: Type 1: Bonafide Messaging with a real Commercial or non-profit(ie political) purpose. This also includes contraband (eg drugs) which is illegal in some or all juridictions, so long as they intend to deliver the illegal goods. This also includes solicited and unsolicited email, though it may be useful to distinguish solicited as Type 1A and unsolicted as Type 1B. Type 2: Bonafide fraudulent activity. Someone is really trying to get your money, but has no intentions of honoring their obligations to the purchase contract. This includes bonafide attempts at identify theft. This is already criminalized by mail fraud and wire fraud, and other laws concerning fraud. Type 3: Annoyance activity. This has no bonafide intention of getting money or even personal information, even though at a casual glance it may appear so. Type 3 is broken into 2 subtypes: Type 3A is a relatively harmless disgruntled person, who is not terribly sophisticated in their abuse, or in hiding their tracks. This type can be handled by warnings or account termination. Besides spam, this type is also involved in small DOS attacks and other unsophisticated abuse. Type 3B is frequently a career criminal using viruses and rooted machines to conduct annoyance, which is frequently just another type of DOS attack, but targeted perhaps at an email address, or perhaps at a domain. This type of attacker is frequently already a career criminal, having broken into many, often hundreds of computers, illegally. This type cannot be dealt with effectively by ISPs, because they are reasonable adept at partially hiding their tracks by crossing organizational boundaries. Usually, the ISP only detects the infected computer, but does not identify or catch the cracker.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I am not talking about email spreading virues. A number of viruses appear to send spam. (not spreading). Sometimes this is autonymous. Sometime it is under control via IRC channel back to the virus operator. Further, it seems that many open proxies are installed by virus. Once the virus has control of the computer, it has or will obtain keys to private keychains, etc. It can do whatever the infected users can do. The number of 40-50 emails per IP figure comes from analysis of spam messages that get by filters, by reviewing how many messages came from the same source. A lot of spam that gets by filters is of this very low volume type. --Dean On Tue, 9 Sep 2003, Shelby Moore wrote: > > Indeed, it seems most of the spam isn't commercial: > >Most of the spam seems to come from viruses, and isn't really selling > >anything. The viruses can use the credentials of the infected user. > >That is "legitimate", until someone reading the email realizes its not and > >complains. These send 40-50 messages per IP, and is hard to detect as > >bulk. > > > This is pseudo-off topic because I already stated below that a viral > signal can be detected differently than a spam signal, unless it > contains no viral data (which would be pointless afaik). I am curious > about your data. Are you refering to emails spreading a virus that > contain viral attachments?? > > It occurs to me that a virus can not spread very fast or effectively if > each infected computer only sends 50 emails, because the infection rate > is probably similar to spam, i.e. < 0.005%. So you would only get 1 new > infection for each 20,000 emails sent, or thus for each 400 infected > computers. It seems the virus would likely die (anti-virus actions) at > that rate of spread. So I must assume you were looking at a very small > sample on internet email and you did not extrapolate??? > > Your answers might be somewhat helpful to me in my work. > > Thanks, > Shelby Moore > http://AntiViotic.com > > >
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Tue, 09 Sep 2003 02:44:21 +0800, Shelby Moore said: > It occurs to me that a virus can not spread very fast or effectively if each > infected computer only sends 50 emails, because the infection rate is probably > similar to spam, i.e. < 0.005%. So you would only get 1 new infection for each > 20,000 emails sent, or thus for each 400 infected computers. It seems the > virus would likely die (anti-virus actions) at that rate of spread. W32/Sobig-F. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> Indeed, it seems most of the spam isn't commercial: >Most of the spam seems to come from viruses, and isn't really selling >anything. The viruses can use the credentials of the infected user. >That is "legitimate", until someone reading the email realizes its not and >complains. These send 40-50 messages per IP, and is hard to detect as >bulk. This is pseudo-off topic because I already stated below that a viral signal can be detected differently than a spam signal, unless it contains no viral data (which would be pointless afaik). I am curious about your data. Are you refering to emails spreading a virus that contain viral attachments?? It occurs to me that a virus can not spread very fast or effectively if each infected computer only sends 50 emails, because the infection rate is probably similar to spam, i.e. < 0.005%. So you would only get 1 new infection for each 20,000 emails sent, or thus for each 400 infected computers. It seems the virus would likely die (anti-virus actions) at that rate of spread. So I must assume you were looking at a very small sample on internet email and you did not extrapolate??? Your answers might be somewhat helpful to me in my work. Thanks, Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> > Information theory says that such things are impossible. One can not >> > construct a spam-free protocol because this is the same problem as >> > constructing a system free of covert channels, which information theory >> > says is impossible. But information theory also says you can optimize signal-to-noise ratio, but only if you know what the characteristics of your signal are. Thus my whole motivation for an unambiguous definition (spam == all bulk email) along the channel and not just a definition at the end points (UBE). >> Nobody cares. Making a roof 100.00% impervious to water molecules >> may be impossible, but that doesn't mean we have to resign to getting >> wet every time it rains. > >People care because when someone comes around saying "you can have a 100% >impervious roof if only you jump through these inconvenient hoops", Who said 100%? Is the problem already reduced to an acceptable S/N ratio for the majority? Is the current S/N ratio a problem for the majority? > we >know that they are wrong, How can you know "they are wrong", even you did not even realize that no one was proposing a 100% solution in this thread?? >"We", meaning the IETF, care, because this is very useful aid to deciding >what to work on. You decline work without even understanding what is being written about a new idea? No one ever proposed 100% solution. I challenge you to find one post where I wrote that. > We know that we need to focus on leak stoppage, not >trying to invent leak-proof protocols. I proposed an way to improve leak stoppage, by defining the signal in the channel and not only at end points. I never proposed a leak-proof protocol. Underlying in what you are saying is you don't support new protocols, because you have a vested interest in existing methods of detection. >We didn't get to the moon by inventing perpetual motion machines, And many people said it was impossible for us to get to the moon. > though >early proposals were based on such machines. We got to the moon by >working on the messy physics of rockets. What about the messy information theory of defining the spam signal for the channel and not just the end points, so that it can be studied and research in the channel and not only in the receiver's mind Have you not realized yet that profound point I am making??? >> > After I first posted this on IETF a while back, someone suggested that >> > covert channels require cooperation, and that spam therefore isn't a >> > covert channel. >> >> Where does this covert channel stuff come from anyway? > >What do you mean? The use of covert here probably meant that making anything 100% covert is impossible in information theory. >> > But this is a simpler way to think about it: Spammers can continue to >> > claim they are legitimate emailers, because they _ARE_ legitimate, so >> > far as we know before they send email. And even so far as we know >> > _before_ someone _READS_ their email. Only after reading their email, >> > and perhaps only after some investigation, can we know for sure that >> > the sender and message is conducting abuse or in violation of their >> > AUP. You exactly stated the problem. We have not defined spam (the signal we need to sample) in terms of the channel. If you define it as ALL BULK EMAIL, then you can actually start some messy science of getting to the moon on spam issue... >> This goes for each individual message, but the spammer's achilles heel >> is that they need to send out incredible amounts of email in order to >> fulfill their objectives, whichever those are. Detecting bulk mail is >> doable, and it shouldn't be too hard to come up with something to >> differentiate legitimate bulk emailing from spam. For instance, we can >> reverse the burden of proof here and only allow know bulk emailers. > >"Detecting abuse" is quite different from making a protocol that can't be >abused. But that is my point: You have to focus on detection. This >doesn't require any protocol changes whatsover. You can not measure a signal if you have not defined it. That is a fundamental concept of information theory. Spam is ambigous in the channel, unless you define spam == all bulk email. How many dozens of times have I written that in this thread! >We are already "only allowing known bulk emailers". Unfortunately, that >doesn't prevent spam. SOBO (statement of blatantly obvious) that you can't filter something if you can't define it. That information theory 101. > Indeed, it seems most of the spam isn't commercial: >Most of the spam seems to come from viruses, and isn't really selling >anything. The viruses can use the credentials of the infected user. >That is "legitimate", until someone reading the email realizes its not and >complains. These send 40-50 messages per IP, and is hard to detect as >bulk. Viruses are a different signal and can be filtered as such, unless they contain no viral da
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 7 Sep 2003, Iljitsch van Beijnum wrote: > On zondag, sep 7, 2003, at 21:45 Europe/Amsterdam, Dean Anderson wrote: > > > Information theory says that such things are impossible. One can not > > construct a spam-free protocol because this is the same problem as > > constructing a system free of covert channels, which information theory > > says is impossible. > > Nobody cares. Making a roof 100.00% impervious to water molecules > may be impossible, but that doesn't mean we have to resign to getting > wet every time it rains. People care because when someone comes around saying "you can have a 100% impervious roof if only you jump through these inconvenient hoops", we know that they are wrong, and don't need to waste time considering how inconvenient the hoops are. "We", meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. > > It is not simply hard. It is impossible, like perpetual motion. > > So when exactly was the earth supposed to stop moving? God didn't make the earth move perpetually. He just made it move long enough. It seems that even God can't solve some problems. We didn't get to the moon by inventing perpetual motion machines, though early proposals were based on such machines. We got to the moon by working on the messy physics of rockets. When someone comes to the NSF and says you can have a perpetual motion machine if only you jump through some very inconvenient hoops, and spend a lot of money, the NSF can save itself the time and money by discarding perpetual motion schemes from its research program. Similarly, information theory allows us to discard some ideas from our research programs. That is why we care. > > After I first posted this on IETF a while back, someone suggested that > > covert channels require cooperation, and that spam therefore isn't a > > covert channel. > > Where does this covert channel stuff come from anyway? What do you mean? > > But this is a simpler way to think about it: Spammers can continue to > > claim they are legitimate emailers, because they _ARE_ legitimate, so > > far as we know before they send email. And even so far as we know > > _before_ someone _READS_ their email. Only after reading their email, > > and perhaps only after some investigation, can we know for sure that > > the sender and message is conducting abuse or in violation of their > > AUP. > > This goes for each individual message, but the spammer's achilles heel > is that they need to send out incredible amounts of email in order to > fulfill their objectives, whichever those are. Detecting bulk mail is > doable, and it shouldn't be too hard to come up with something to > differentiate legitimate bulk emailing from spam. For instance, we can > reverse the burden of proof here and only allow know bulk emailers. "Detecting abuse" is quite different from making a protocol that can't be abused. But that is my point: You have to focus on detection. This doesn't require any protocol changes whatsover. We are already "only allowing known bulk emailers". Unfortunately, that doesn't prevent spam. Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The viruses can use the credentials of the infected user. That is "legitimate", until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. But when added up over a lot of IP addresses, is quite obviously annoying. > > It is not immune to spam, though it distributes spam and other > > broadcast messages much more efficiently than typical email systems. > > Ouch! :-) > > Fixable with authentication. No, that's the point. It isn't _fixable_ with authentication. It isn't fixable at all. It is only "fixed" when the spammer loses his account. Then the spammer gets a new account. So it isn't really fixed. So we are always going to be playing a game of whack-a-mole. That cannot be avoided by altering the protocol or the authentication scheme (information theory proves this). So it is useful, then, to work on ways of detection, and improve our whack-a-mole skills. Altering protocols and authentication is a waste of time. --Dean
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Main arguments made thus far and my retorts. A1: Any one who tries to work on anti-spam is a "Kook". R1: Illogical A2: Too difficult for legitimate bulk senders to implement and support, and "especially" mailing lists. R2: Many, if not most, mailing lists are already provided in "pull" www format. Since they already implemented the complexity to pump incoming emails into www, it is trivial in comparison to cc: a copy to a POP account. A simple .procmail script could be written in 5 minutes. A3: Spammers will use mailing lists to send. R3: They already do. Receivers can opt out of mailing lists (before and after my proposal). Without my proposal, they can opt-out of all bulk email, which includes spam and legitimate bulk email, and they can selectively whitelist to opt-in (I showed how this whitelisting can be spoofed by a spammer). With my proposal, they can additionally opt-out out all spam == which is now all bulk email, and selectively opt-in (via "pull") to mailing lists (which is a concurrent operation with subscribing, not a separate subvertable whitelist). With my proposal, receivers will get less spam, because their email address is only selectively opted-in, instead of open to all spammers. Additionally, with the inherent "pull" delay, mailing lists can effectively remove spam from the "pull" server as it is discovered, well before most receivers "pull" it. Additionally, mailing lists already have mechanisms in place to defeat incoming spam and will be developing more, given their advantage of requiring membership (subscription). So all in all, under my proposal spammers will be less successful. A4: Status quo is better risk. R4: Timeline curves project spam will keep increasing and eventually most email will be spam. Unless effective filters can be done and maintained with the status quo paradigm, you will eventually have 1 legitimate email for every 10s or 100s of spams. I know people who have this scenario already. At that point, the legitimate bulk email isn't getting read whether it is filtered actively or inherently (burried in spam). Without my proposal and with this eventual outcome, receivers will be forced to use "pull" for mailing lists (e.g. www archives) while their general email remains broke. If you think current filters are effective, then why are so many receivers complaining about spam? I know there are filters which work somewhat, but either their cost model can apparently not scale to all receivers (e.g. BrightMail because they use humans to help filter), or the filter can be subverted with varying content or whitelist mining (e.g. DCC), or the filter can be subverted by adapting Bayesian footprint, or the filter can be subverted by whitelist mining and filter causes big delays in legit email (e.g. email-back whitelisting verification), etc... The point is legitimate bulk email is going to end "pull" whether by design or by failure of general email. I prefer by design. A5: Receivers will not "pull" legitimate bulk email R5: They will if there is no other way to get their email. See R4 above. By design or by failure of general email, the problem is going to collapse either way into "pull"ing legitimate bulk email. I prefer by design. A6: POP is not ubiqutous enough. R6: Use any and as many "pull" protocols as you want to serve your readers. I'd say POP and www are sufficient, but use as many as your receivers want. Let your market decide what you provide. A7: Some people in this list are against because they have vested interest in status quo for mailing lists R7: Without logical basis on the internet as a whole (this is IETF), that would not be a sound engineering basis to stifle an idea. A8: Usenet already exists R8: Use Usenet to provide pull if that is what your receivers want. I think most would prefer to send and recieve in their email client, so I think POP would be more popular. A9: Separating legitimate bulk email into "pull" will not cause additional enforcement against remaining spam == all bulk email R9: There is going to be continuing increase in enforcement whether you implement my proposal or not. Either active enforcement or inherent (lost in pile of spam). There is a fundamental science to this. It is called information theory and one of the theorems is that as you increase noise, then unless you can filter the noise, receiving original signal becomes less reliable. The response rate of spam at < 1% (figure I've seen is usually < 0.003%) means it is noise. Reading (receiving) of all email will get more and more unreliable, as spam increases. Just imagine the bandwidth it will take to download it, store it, process it, not to mention try to filter and read it. Spammers will get more and more savvy at subverting filters, as filters become more and more successful and popular. Spammers will increase spam, the more that filtering increases. It is a race to the saturation poin
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> However, what is the harm in making an RFC and then find out if enforcers >> will enforce?? > >you appear to presume that you can get consensus support for such a plan >from within IETF. No, no. I try to never beg. I came here to make a public proposal and some points for the purposes of public record. I never expected any one to really do any thing on this proposal now. But by placing the idea into public domain now, as the rate of spam approaches the asymptote of 100%, then mailing list administrators can come back to this idea as a way to save themselves. Actually at that point, they won't need the RFC or STD, because all bulk email will be blocked and they will be forced to just go straight to offering a "pull" solution (whether it be web, usenet, pop or whatever) so recievers can still get their mailing list messages amongst the 10,000 spams per day. I am not sure exactly how long this will take, but without another solution or paradigm shift, I figure 2 - 5 years tops. > even if you could get such support (which you cannot) How do survey their opinion?? >note that there's no enforcement of IETF's other opinions, even in cases where >failure to adhere to IETF standards costs billions of dollars every year. why >would this case be any different? Because enforcers want to do something and will be under increasing pressure to do something against bulk email. >there are a lot of ways to solve the spam problem that will work if "everybody >does it my way." I have seen none. Email signing won't work. No amount of from Randy Bush will change that. Changing SMTP won't work long term, etc.. > so far, nobody has figured out how to impose their will on >the rest of the net. It is not my, your, or IETF will, but the votes of 500 million (through their actions against spam) that will impose. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I am nearing the end of my allow time to respond, so if I do not respond in future, it
doesn't mean I agree :)
below...
>> 2. Regarding additional burden on *legitimate* bulk message *senders*:
>>
>>a. These senders are much, much fewer than the # of receivers suffering
>>from spam. Any incremental cost on the few is justified.
>
>yeah, well good luck convincing *them* of that. especially when you can't
>even convince *us* of that.
You ("us" not including me and perhaps some others) are apparently "them", so I see no
need for the word "especially".
Further, there is no need to convince the minority. Majority wins. If enforcers
successfully block all bulk email with my proposal (as an internet STD) sanctioning
it, then receivers (the minority) will dictate the outcome.
I assert that even if 50% spam is not bad enough to make the 500 million rise up, just
wait until it is 90+%, which again I assert won't be too long from now.
>>b. The senders are already quite resource savvy, else they would not be
>>sending *legitimate* bulk (in statistical significance) messages. I
>>doubt the incremental cost is significant to cause failure.
>
>let's see. it doesn't cost the spammers much to send bulk mail...
Oh it costs spammer something. They are definitely more resource savvy than the
majority, which was my point. The burden on the minority is less important, if the
majority is making big gains from the proposal.
>so why
>do you assume that the costs are significant to legitimate senders?
They have to maintain mailing list servers, mailing list databases, manage support for
subscribers, handle incoming spam, etc.. All things which an average (the majority)
individual user of email doesn't have to do (or won't bother to do).
>>c. And compare this burden to the burden they have with dealing with
>>spam, which would be eliminated by this proposal. If spam isn't
>>eliminated, then they need not adopt the proposal.
>
>most of the burden of dealing with spam is on the recipient, not the sender.
Agreed currently. But not once my proposal succeeded, which is a correction I made
after making this point #c above. Point #c is invalid. Please remove it.
>apparently you think that legitimate senders should pay for the cost of
>lessening the burden on recipients from illegitimate senders.
Insert "bulk" after legitimate and illegitimate, then I agree with your statement.
The reason is because *legitimate* bulk message senders are going to pay one way or
the other soon. It is like the old Fram oil filter commercial in USA, "Either pay me
now (replace filter) or pay me later (replace motor)"
As the rate of spam approaches 90% or 99%, receivers are going to opt to block all
bulk email, irregardless of whether my proposal is in place. They won't have any
other choice, else quit using email.
Now that is profound prediction!
> why do I think
>you're not likely to get much support for that view from the senders?
Oh yes to be expected that human nature tends to manicure (defend) their own feet
instead of watching where they are going and what they will soon bump into.
Shelby Moore
http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> so far, nobody has figured out how to impose their will on > the rest of the net. thankfully > > Keith > > sleekfreak pirate broadcast world tour 2002-3 live from the pirate hideout http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> However, what is the harm in making an RFC and then find out if enforcers > will enforce?? you appear to presume that you can get consensus support for such a plan from within IETF. even if you could get such support (which you cannot) note that there's no enforcement of IETF's other opinions, even in cases where failure to adhere to IETF standards costs billions of dollars every year. why would this case be any different? there are a lot of ways to solve the spam problem that will work if "everybody does it my way." so far, nobody has figured out how to impose their will on the rest of the net. Keith
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Excuse me, it is a valid issue that spammers will try to pipe through mailing lists (legitimate bulk email) to avoid *BE enforcers. Mailing list administrators will continue to carry this burden and probably more so under my proposal. Thus yes I agree that authentication of incoming to "pull" servers is a major consideration. However unlike with current paradigm, the receiver can then opt-out of the spam. Thus darwinism will force effective solutions, because those mailing lists that are successful will have more receivers than those who are not (because the receivers how power of opt-out of spammy lists unlike now where they can not opt-out of spam email because their legit bulk email not unbundled). And remember that commercial legitimate bulk email is not effected by incoming pollution issue. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> As each individual news article is piped through a relatively small >> number of servers in the "core" of the distribution system, it becomes >> relatively easy to blacklist known offenders. That is, if they are >> recognizable as such. > >No way. My proposal does not depend on authentication of what is incoming to the "pull" servers, because by definition it enables the enforcers to filter *BE (all bulk email) which (by the proposal) is defined to be all spam. This is a non-issue relative to my proposal. If you are really paranoid, use https web page to take incoming from authenticated users. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>As each individual news article is piped through a relatively small >number of servers in the "core" of the distribution system, it becomes >relatively easy to blacklist known offenders. That is, if they are >recognizable as such. This is where the authentication comes in. The >tricky part is optimizing the time/difficulty it takes to blacklist vs >that of obtaining a new identity. Even though this is a good point, I'd prefer to stay off the discussion of authentication, because the proposal need not depend on it, as you point out below... >Also note that with usenet news, unlike email, it is possible to remove >spam that has entered the system, limiting the audience that sees the >message and thus the effectiveness of spamming. Also bear in mind my point that the whole point of moving legitimate bulk messaging to "pull" is so that spam can be dealt with unambiguously by enforcers on bulk email. So spam (all remaining *BE) gets reduced by the paradigm switch to "pull" for legitimate bulk messaging. Since I assume mailing lists would still use email as incoming source primarily, then incoming spam is reduced. You could certainly switch to a authenticated interface (e.g. https web page) for incoming mailing list traffic, but I think that is unnecessary to my proposal. Also remember that some portion of legitimate bulk messaging is legitimate (meaning that receivers want it) corporate bulk mailings. They would also be forced to the "pull" paradigm, but not their normal single messaging, such as the receipt for what you buy on Amazon, only their bulk email. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Want to elaborate on a few points I made:
1. Regarding whether POP is ubiquitous enough to be the mechanism for "pull", I think
that is just details. The overall proposal is to use "pull" (in what ever form)
instead of "push" email for *legitimate* bulk message delivery.
2. Regarding additional burden on *legitimate* bulk message *senders*:
a. These senders are much, much fewer than the # of receivers suffering from spam.
Any incremental cost on the few is justified.
b. The senders are already quite resource savvy, else they would not be sending
*legitimate* bulk (in statistical significance) messages. I doubt the incremental
cost is significant to cause failure.
c. And compare this burden to the burden they have with dealing with spam, which
would be eliminated by this proposal. If spam isn't eliminated, then they need not
adopt the proposal. It is a self-feeding ("chicken and egg" go together) proposition.
3. Regarding additional burden on *legitimate* bulk message *receivers*, here is I
think your best argument for significant pain in adoption. Of course it was also
painful during the browser wars. Pain of transistion is not an unknown experience on
the internet. And compare this potential pain of adoption to the very significant
pain and cost of spam (which is getting only worse).
4. Regarding whether spam is significant to the economy of internet (and thus has
powerful vested supporters), if it is possible that an activity that gets on average
less than 0.005% response rate (or even if you assume 1%) is significant compared to
ALL other internet activities combined, then that is sad statement about the internet
economy. I am very confident that is not true assumption. And compare this to the
massive cost of spam. There is link I could dig which estimates that the yearly cost
of spam to business is $9 billion. Do I need to dig it up or can you Google for it?
Shelby Moore
http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Iljitsch van Beijnum <[EMAIL PROTECTED]> wrote: > On maandag, sep 8, 2003, at 00:08 Europe/Amsterdam, Johnny Eriksson > wrote: > > >>> It is not immune to spam > > >> Fixable with authentication. > > > no. > > As each individual news article is piped through a relatively small > number of servers in the "core" of the distribution system, it becomes > relatively easy to blacklist known offenders. That is, if they are > recognizable as such. No way. If this was indeed the case, it would already have been handled by blacklisting known AS numbers / prefixes in the routing system. This is not the case. See below. > This is where the authentication comes in. This is where layer eight and above comes in. There are enough people that profits from this that us low-lifes that want spam to stop just don't count if you want to play by the rules. I'm all for the kneecapping proposal by citizen V.K. --Johnny
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
This is getting way off topic. >One of the other things you see to be handwaving a bit about is >the notion of handing out user IDs, passwords, and other >credentials to mail accounts to people so they can "help" with >spam (or other problems). My proposal has nothing to do with IDs so let's just drop that line of discussion or respond with a different subject to move it to a different thread. > Sure, I can find a web-based >something-or-other to access my POP3 mailbox (if I had one). Agreed. >the web site listed in your >signature line seems to mostly run people around in circles My web site and businesses have nothing to do with this proposal. The proposal is to define that *legitimate* bulk email must be sent and received by "pull" instead of by "push". So that all remaining bulk email is spam. This changes spam from "Spam == UBE" (an ambiguous definition for enforcers) to "Spam == *BE" (UNambiguous definition for enforcers). How *BE gets dealt with by enforcers is not related to my businesses or web site. Enforcers is a widespread term meaning any one who will enforce the proposed RFC against spam. FYI: (completely off topic so please don't respond to list on this point) The reason the web site runs in circles is it isn't finished yet, nor ready to be released to public at large yet. It should be completed in a few more days. Shelby Moore http://AntiViotic
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On maandag, sep 8, 2003, at 00:08 Europe/Amsterdam, Johnny Eriksson wrote: It is not immune to spam Fixable with authentication. no. As each individual news article is piped through a relatively small number of servers in the "core" of the distribution system, it becomes relatively easy to blacklist known offenders. That is, if they are recognizable as such. This is where the authentication comes in. The tricky part is optimizing the time/difficulty it takes to blacklist vs that of obtaining a new identity. Also note that with usenet news, unlike email, it is possible to remove spam that has entered the system, limiting the audience that sees the message and thus the effectiveness of spamming.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Keith, IMHO you started an excellent line for further debate (and not just because we have the same last name :). It would be nice to see debate from both sides so that pros and cons could be fully explored. I am not sure I am the one to carry the debate to extreme end (due to time constraints), but I will initiate the counter argument. I came here to make a proposal. It is not a proposal I would champion to the end game alone. The feedback is valuable (hopefully to community-at-large as well), whether the proposal florishes or not. >> What you are saying IMO, is that you can't force bulk emailers or spammers >> to use opt-in. > >Let's be even clearer. What's being claimed is that you can't force bulk >emailers to send their email via "pull" technology (whether this means >providing their own POP servers, IMAP servers, NNTP servers, web servers, >whatever) while everyone else can still use "push". "Enforcers" will force the adoption. "Enforcers" being Hosts, ISPs, legislatures, judiciaries, software, etc.. Once you define that *legitimate* bulk email is only delivered by "pull", then enforcers can start to filter bulk email. This will force everyone to get their *legitimate* bulk email via "pull". Else they will not get it. If you have a choice between receiving your *legitimate* bulk email via "pull", or not receiving it, what are you going to do? Now below you argue that enforcers will not enforce. To me that is more viable argument, one that needs to be debated. However, what is the harm in making an RFC and then find out if enforcers will enforce?? more below... > And the question isn't >really whether bulk mail can be statistically distinguished from non-bulk >mail (since that's really just a matter of whether you can get people to >adopt a definition of "bulk" in terms of externally visible traffic >properties) - the question is whether you can enforce that rule. Agreed. Good point about benefits (real results) of agreeing on definitions. That is in essence why I am making this proposal. I am proposing we make a definition of spam which is both agreed upon and not ambiguous to enforcers. Simple as that. In my mind, the thing to consider is what is the cost of making this definition compared to what is the benefit?? To me, that is one of the major aspects that need to be debated. Economics (in the larger sense of "economy" as efficiency or survival/evolution, not just financial systems) *always* rules the outcome. >IMHO - most recipients don't want to get their mail that way (and many of >the deployed user agents don't support it), Most people already get their email through POP, even if there is a web-based client on top of it. We already went through the discussion that Hotmail and Yahoo have the ability to access POP accounts. They wouldn't have the feature if people aren't using it. > most senders don't want the >increased burden of providing POP/IMAP/NNTP/web servers Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Non-bulk email and spam does not change it's paradigm. Spam is dealt with by enforcers. Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Just making that very clear. *Legitimate* bulk email senders already have a burden of maintaining mailing list databases, maintaining a subscribe and unsubscribe support, handling bounces, etc...s > and the necessary >customer support, I assert there will be less or similar hassles with supporting a "pull" paradigm for *legitimate* bulk email than all the hasslings of teaching newbies how to subscribe and correctly interface with Majordomo mailing list. I do not think it is a big difference. And if you are protecting the mailing list administrators to save spam, then that seems like a bad set of priorities for internet email as a whole. > and there are enough ISPs that derive significant revenue >from selling bandwidth to spammers that it would be extremely difficult to get >them all to enforce this. This is wrong. This would be saying that the economy of spam is significant compared to the internet economy as a whole. ISPs are losing $ big time to spam. > In short: nobody has sufficient incentive to adopt >this. Maybe "nobody" in this list, but let this thread sit in Google for a while and see what might snowball over time :) >Of course there's nothing wrong with defining another way to distribute bulk >mail that people can use if they wish. Great. That is what I said above. Why not see what the enforcers will do? They can not do any thing now, because their hands are tied to ambiguous definition of spam as UBE. They need spam == *BE in order to act. > If it works well, some people will >use it. The stretch is to insist that everybody do it this way. No. If it work
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
--On Sunday, September 07, 2003 17:07 -0400 "vinton g. cerf" <[EMAIL PROTECTED]> wrote: >> I understand but that was not my point. My point is that you >> can put a web-based interface on top of your POP account to >> access it any where. You still have a POP account which you >> are accessing any where if that is what you want. The >> web-based interface is just another form of an email client. > > that's different - what you said was as quoted above. I agree > that if you design the web server properly, you can use a web > interface, but you run the risk that with this design, you may > never be able to pull the email later, POPStyle, into your > computer. Although it is theoretically possible, using POP > (rather than IMAP) to leave the mail on the server until you > pull it again with POP, many servers appear to clear out the > mail after POPing it. I think John Klensin made that > observation in an earlier exchange. I may not have, but, to get it on the table, many ISPs who offer "free" POP3 accounts have decided that, in order to control costs, they need to do one (or more) of three things: (i) Impose total-quantity-of-mail limits on mailboxes (ii) Charge for storage, at least above a certain, fairly small, amount. (iii) Prevent the user from using the mail store for long-term storage by deleting messages that have been downloaded, whether the user issues an explicit POP3 "delete" command or not. If the third model is adopted by a POP3 provider, then reading through the mailbox looking for spam will presumably be 100% effective in eliminating spam... because it will be 100% effective in eliminating everything. That is probably not the desired result. One of the other things you see to be handwaving a bit about is the notion of handing out user IDs, passwords, and other credentials to mail accounts to people so they can "help" with spam (or other problems). Sure, I can find a web-based something-or-other to access my POP3 mailbox (if I had one). But whether I can find one I trust as much as I trust my mail store and the existing mechanisms for reaching it is an open question. If I understand what you are proposing --you have actually been a little vague, and the web site listed in your signature line seems to mostly run people around in circles-- you require that I supply you sufficient credentials to read --and potentially delete and maybe alter-- my mail.I assume there are reasons why I should trust you that much, but, independent of anything else, your establishing those would be, I think, an interesting problem. john
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
You can get mail no matter where you are with a POP account also. >>> >>>shelby, that's actually not true. If you have an enterprise email service >>>that requires access to a VPN and the internet service you access it with >>>(e.g hotel room ethernet) has a bad firewall configuration, you may never >>>get to the mail. I speak with personal experience - the hotel I am in right >>>now has screwed up its firewall. I ended up having to find an 802.11 hotspot >>>to get to my email. >> >>I understand but that was not my point. My point is that you can put a >web-based interface on top of your POP account to access it any where. You >still have a POP account which you are accessing any where if that is what >you want. The web-based interface is just another form of an email client. > >that's different - what you said was as quoted above. In all due respect, I had written: You can get mail no matter where you are with a POP account also. You are writing about different issues of which email client is ideal for each scenario and what the feature set needs to be. My statement is still true. Programmers can massage (program) the email client as needed, then my statement is never false. I assert there are already email clients to access POP from most any where, even if they aren't well known. > I agree that if you >design the web server properly, you can use a web interface, but you run the >risk that with this design, you may never be able to pull the email later, >POPStyle, into your computer. Although it is theoretically possible, using >POP (rather than IMAP) to leave the mail on the server until you pull it >again with POP, many servers appear to clear out the mail after POPing it. I >think John Klensin made that observation in an earlier exchange. > > >>The point is that you don't need to use a web-based email without an >underlying POP account in order to access email from any where. There are >even places where HTTP web-based interface won't work (e.g. cell phone) and >so you need to use a different form of email client to access. Still you >can have an underlying POP account that mail is being drawn from. > >see above.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 04:24 AM 9/8/2003 +0800, Shelby Moore wrote: >>At 11:51 AM 9/7/2003 +0800, you wrote: >>>You can get mail no matter where you are with a POP account also. >> >>shelby, that's actually not true. If you have an enterprise email service >>that requires access to a VPN and the internet service you access it with >>(e.g hotel room ethernet) has a bad firewall configuration, you may never >>get to the mail. I speak with personal experience - the hotel I am in right >>now has screwed up its firewall. I ended up having to find an 802.11 hotspot >>to get to my email. > >I understand but that was not my point. My point is that you can put a web-based >interface on top of your POP account to access it any where. You still have a POP >account which you are accessing any where if that is what you want. The web-based >interface is just another form of an email client. that's different - what you said was as quoted above. I agree that if you design the web server properly, you can use a web interface, but you run the risk that with this design, you may never be able to pull the email later, POPStyle, into your computer. Although it is theoretically possible, using POP (rather than IMAP) to leave the mail on the server until you pull it again with POP, many servers appear to clear out the mail after POPing it. I think John Klensin made that observation in an earlier exchange. >The point is that you don't need to use a web-based email without an underlying POP >account in order to access email from any where. There are even places where HTTP >web-based interface won't work (e.g. cell phone) and so you need to use a different >form of email client to access. Still you can have an underlying POP account that >mail is being drawn from. see above. v Vint Cerf SVP Technology Strategy MCI 22001 Loudoun County Parkway, F2-4115 Ashburn, VA 20147 703 886 1690 (v806 1690) 703 886 0047 fax [EMAIL PROTECTED] www.mci.com/cerfsup
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Iljitsch van Beijnum <[EMAIL PROTECTED]> wrote: > > It is not immune to spam, though it distributes spam and other > > broadcast > > messages much more efficiently than typical email systems. > > Ouch! :-) > > Fixable with authentication. no. --Johnny
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>At 11:51 AM 9/7/2003 +0800, you wrote: >>You can get mail no matter where you are with a POP account also. > >shelby, that's actually not true. If you have an enterprise email service >that requires access to a VPN and the internet service you access it with >(e.g hotel room ethernet) has a bad firewall configuration, you may never >get to the mail. I speak with personal experience - the hotel I am in right >now has screwed up its firewall. I ended up having to find an 802.11 hotspot >to get to my email. I understand but that was not my point. My point is that you can put a web-based interface on top of your POP account to access it any where. You still have a POP account which you are accessing any where if that is what you want. The web-based interface is just another form of an email client. The point is that you don't need to use a web-based email without an underlying POP account in order to access email from any where. There are even places where HTTP web-based interface won't work (e.g. cell phone) and so you need to use a different form of email client to access. Still you can have an underlying POP account that mail is being drawn from.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On zondag, sep 7, 2003, at 21:45 Europe/Amsterdam, Dean Anderson wrote: Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. It is not simply hard. It is impossible, like perpetual motion. So when exactly was the earth supposed to stop moving? After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. This goes for each individual message, but the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. However, I looked at your proposal, and it appears that you are trying to create a "pull" mechanism rather than a "push" mechanism for message delivery . This paradigm has already been implemented as "Usenet News". My point exactly except that usenet doesn't have to be significantly more "pull" than email (where you need your client to check for new mail as well). It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. Ouch! :-) Fixable with authentication.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. It is not simply hard. It is impossible, like perpetual motion. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. I have recently discussed the issue at length with some PhD's and other very smart people from one of my old companies, and we determined after a great deal of discussion that covert channels (and similar concepts like side channels and information leakage, and other terms used in information theory literature) don't require cooperation as a necessary condition. The term "Covert Channel" is mostly used in liturature analyzing operating systems. However, the general concept is referred to in non-OS liturature as "Side Channels" or "Information Leakage", or "Sneaky Channels". The concepts are essentially the same. Indeed spam can be modeled as a communication channel that is multiplexed with email, which is itself multiplexed with TCP, which is multiplexed with IP, which may be multiplexed over a variety of physical media. It was also realized (towards the end) that spam is in fact a cooperative covert channel, since the reader cooperates when the receive the spam and open it for reading. Only _after_ reading the spam message are some people offended, but this is of no consequence to the application of information theory because the communication has already taken place. All they can do, having now detected the abuse, is try to suppress further abuse that is similar. But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. Some might ask about blocked abuse. Subsequent abuse is blocked because it is similar to previous abuse (by IP address with blacklist), or by content with content filters. We can anticipate some likely abusive content, but cannot identify all abusive content in advance. However, I looked at your proposal, and it appears that you are trying to create a "pull" mechanism rather than a "push" mechanism for message delivery . This paradigm has already been implemented as "Usenet News". It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. However, bandwidth consumed by spam email (indeed bandwidth consumed by _ALL_ email combined), is still minor, so we are really not in need of a more efficient means of distributing spam, or email for that matter. --Dean On Sat, 6 Sep 2003, Shelby Moore wrote: > Request for opinions on whether to creating a working group or publish > the following idea as an internet draft? > > Spam is big problem that is getting worse. BrightMail.com (which claims > to process 10% of world's email) claims that the percentage of spam out > of all email has grown from 16% in Jan. 2002 to 50% in Aug. 2003. > > A fundamental unsolved problem of doing any thing about spam, is there > is currently no unambiguous definition of spam as an enforceable > internet standard. This has been architectually impossible to define > because the receiver is the subjective determinant of which bulk email > is solicited and which is spam (UBE). > > ISPs, Hosts, legislators, judiciaries, and even anti-spam software, have > a fundamental problem in that definition of spam as UBE is currently > architectually unenforceble due the fact that subjective determination > of "unsolicited" current happens after the email has been delivered to > the receiver. > > My idea is to create an internet draft, RFC, and hopefully internet > standard, that would define a simple architectual paradigm for > legitimate bulk email that unambiguously separates it from spam (UBE). > > Simply define that legitimate bulk distribution of email should be done > by mechanism of each bulk distributor providing a public POP3 (and IMAP) > account or server, rather than sending the email directly. > > In the case of a public distribution (e.g. most direct email and mailing > lists), a POP3 (and IMAP) account of user "anonymous" with password > "none" would suffice. In the case of private dissemination (private > mailing lists), a POP3 (and IMAP) server with individual accounts could > be provided. > > The elegance of this paradigm is that users then control the > opt-in/opt-out database, by configuring their email client to POP email > from only the bulk POP accounts they wish to subscribe to. > > The effort to support this
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> > I'll be back here in this list later (probably a year from now) when your needs have > changed to a more dire state regarding email. > Thank you for playing. > > sleekfreak pirate broadcast world tour 2002-3 live from the pirate hideout http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 12:56:04 +0800 Shelby Moore <[EMAIL PROTECTED]> wrote: > > What you are saying IMO, is that you can't force bulk emailers or spammers > to use opt-in. Let's be even clearer. What's being claimed is that you can't force bulk emailers to send their email via "pull" technology (whether this means providing their own POP servers, IMAP servers, NNTP servers, web servers, whatever) while everyone else can still use "push". And the question isn't really whether bulk mail can be statistically distinguished from non-bulk mail (since that's really just a matter of whether you can get people to adopt a definition of "bulk" in terms of externally visible traffic properties) - the question is whether you can enforce that rule. IMHO - most recipients don't want to get their mail that way (and many of the deployed user agents don't support it), most senders don't want the increased burden of providing POP/IMAP/NNTP/web servers and the necessary customer support, and there are enough ISPs that derive significant revenue from selling bandwidth to spammers that it would be extremely difficult to get them all to enforce this. In short: nobody has sufficient incentive to adopt this. Of course there's nothing wrong with defining another way to distribute bulk mail that people can use if they wish. If it works well, some people will use it. The stretch is to insist that everybody do it this way.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>And in fact, unless >you're able >to make the POP check frequency less than the posting frequency, you'll lose. That is quite an easy optimization to do. Study Probability Theory and Statistics (a higher level math class). > >> Whitelisting can be subverted by spammers: >> >> http://www.cnn.com/2003/TECH/internet/09/01/spam.chainletter/index.html > >> "If I were a spammer, I'd be working very hard to perfect this technique," >he said..." > >And your proposal does nothing to stop this, since it won't look like bulk >mail, >it will look like personal mail. *plonk* wrong again. That about roughly 20 *plonks* in a row for you. You have not made one factually correct statement yet in this thread. The bulkness as measured by ISPs and Hosts (those who can enforce the law in real-time) is done by looking at statistics on IPs and packets. It has nothing to do with the mutable fields of email, such as headers or content. If you don't believe that email can be reliably statistically profiled by the source ISP, then go Google before you respond. >> Yes we probably do. Just because the DCC can not measure bulk email reliably >> doesn't mean Hosts, ISPs, and other software can not. BrightMail already >is ( >> just signup for an Earthlink account and try really hard to get some >spam), and >> I will also be probably be demonstrating something soon. > >So why do we need to move off mailing lists, when the problem is solved? Because I wrote "bulk". I did not write "legitimate bulk". Current algorithms have no way to differentiate the two. Remember that was the whole point of this proposal. *plonk* you don't even remember what the subject of this thread is nor do you read my posts carefully. >> It is has nothing to do with what spammers will or will not do. It has to do > >Actually, it has everything to do with what they will or wont do. But that is the reaction, not the action. The action (which is first step needed) is done by enforcers... > >> with what Hosts, ISPs, etc are currently prevented from doing. Since they >can >> not determine what is spam, they can not enforce any law. > >And separating out mailing list traffic doesn't change matters, really. *plonk* again. I did not write "separate mailing list traffic". I proposed to "differentiate legitimate bulk email from Spam (UBE)" (see the Subject line above). And if you can separate the legitimate bulk traffic from the UBE traffic, then the existing enforcers (BrightMail, DCC, Hosts, ISPs, etc) can take action against all remaining bulk which is then all UBE. >You're left with a lot of non-bulk high-volume business e-mail (yes, you WANT >this to get through, otherwise Amazon doesn't have a way to tell you easily >about a problem with your order, or similar), a lot of person-to-person mail, Agreed all non-bulk. >and a lot of spam pretending to be one or the other of the above. Which is bulk and can now be attacked by enforcers because the legitimate bulk emailer will not be harmed in process, because it is separated. > The only >thing you've cut out is spam to mailing lists *plonk* again We've moved the legitimate bulk email to "pull". The spam (UBE) gets attacked and reduced by existing enforcers that know how to detect bulk. And then non-bulk is untouched. That is the ideal solution. Nothing orthogonal at all. You think it is orthogonal because you are not visualizing how the architecture change actually works to reduce (or eliminate) spam. >Given that so much spam is already breaking some law, why do you think "the >ISPs >could enforce a NEW law" would make any difference? Example. If I have a customer I am Hosting who is sending bulk email (which I as Host or ISP can measure statistically), but that customer claims the email is legitimate. There is nothing I can do in real-time, automated way to refute his claim. The cost for me to attack the spam is greater than the cost I will save. Therefor I punt and say "spam vs. legit bulk email is ambiguous". Whereas if I have a clear rule that all bulk is spam (UBE), then I can make more profit by setting up automated, real-time, statistical blocking. Now I drive the cost of spamming higher, eventually to a point that it is no longer profitable. Not just one enforcer or one type of enforcer, but all the examples I gave in my posts take in unison. If you want to understand any likely action, always study the economics. >> Yes my proposal depends on that fact. Once you have the legitimate email >> separated from the spam architecturally, then you can effectively increase >the >> cost of spamming to the point it is a non-economically viable activity. > >This would be wonderful, except what you're separating isn't legitimate/spam, >it's mailing list/non mailing list. The two are orthogonal concepts. *plonk* again I do not how you got so confused. Read above. >*plonk* Somebody wake me up if Shelby starts addressing
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 14:02:30 +0800, Shelby Moore said: > POPing once (one list mailing) versus processing one email with zillion RCPT > TOs (one list mailing) is not a very big cost difference. One might be > slightly less than the other and we really can't say which one, but it is > irrelevant because the difference is insignificant. When you compare a successful SMTP to a successful POP, yes. > Actually it is more likely that when they POP they will get several messages > at once, so less cost than catch several SMTP emails. Actually, the most likely is "POP once an hour for a once-a-week posting, and you've totally blown 167 pointless transactions". And in fact, unless you're able to make the POP check frequency less than the posting frequency, you'll lose. > Whitelisting can be subverted by spammers: > > http://www.cnn.com/2003/TECH/internet/09/01/spam.chainletter/index.html > "If I were a spammer, I'd be working very hard to perfect this technique," he > said..." And your proposal does nothing to stop this, since it won't look like bulk mail, it will look like personal mail. From people you know and trust, and everything. > Yes we probably do. Just because the DCC can not measure bulk email reliably > doesn't mean Hosts, ISPs, and other software can not. BrightMail already is ( > just signup for an Earthlink account and try really hard to get some spam), and > I will also be probably be demonstrating something soon. So why do we need to move off mailing lists, when the problem is solved? > It is has nothing to do with what spammers will or will not do. It has to do Actually, it has everything to do with what they will or wont do. > with what Hosts, ISPs, etc are currently prevented from doing. Since they can > not determine what is spam, they can not enforce any law. And separating out mailing list traffic doesn't change matters, really. You're left with a lot of non-bulk high-volume business e-mail (yes, you WANT this to get through, otherwise Amazon doesn't have a way to tell you easily about a problem with your order, or similar), a lot of person-to-person mail, and a lot of spam pretending to be one or the other of the above. The only thing you've cut out is spam to mailing lists - and if you can solve the OTHER two flavors above, then this third is a non-issue anyhow. Given that so much spam is already breaking some law, why do you think "the ISPs could enforce a NEW law" would make any difference? >From another note: > Yes my proposal depends on that fact. Once you have the legitimate email > separated from the spam architecturally, then you can effectively increase the > cost of spamming to the point it is a non-economically viable activity. This would be wonderful, except what you're separating isn't legitimate/spam, it's mailing list/non mailing list. The two are orthogonal concepts. *plonk* Somebody wake me up if Shelby starts addressing the actual problem, rather than an orthogonal non-problem pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
There are some false characterizations that I can not leave unrefuted in public... >Mr. Moore contacted to ask me to sign a non-disclosure agreement No I contacted you to ask if I could have access to the public IP data that is shared between all DCC servers. I told you I would be using it to validate or invalidate an anti-spam algorithm I have devised. And I told you that I did not want to reveal the algorithm to you. You were skeptical so I said that the only way I could reveal more was under NDA. Then you flipped out and even refused my further requests for clientID and password, which is (or was) advertised from your site to be a publicly available resource. Go figure! > so >that I might learn how to make the DCC "effective" based on his new >"intellectual property." I said that my algorithm could complement/improve the DCC by providing automatic whitelisting. You interpreted that statement incorrectly. I also said that the DCC must already be effective else you would not have so much market share. > He was also interested in buying copies of >DCC data. I said that if you weren't willing to give me the data you share publicly for free, then I would be willing to contribute to the DCC to pay for any effort to gather the data I needed. You apparently misinterpreted my considerate and respectful demeanor as some attempt to buy you off. > My responses started cool and eventually became as "overtly" >negative as I could without calling him names. Agreed. > I told him that he >had disclosed nothing of his intellectual property to me, Agreed. You signed no NDA, so why would I. I did not need to. I only needed to get some statistical data for testing an algorithm. I offered that it might help in the overall fight against spam. You could have been more respectful and cordial in declining my request. Given I was only asking for data that is already public, it still makes no sense to me. But no matter, I have moved on and devised a better plan to get data. > that there >is no plausible chance we might ever do any business, I never wanted to do business and I told you that. I only wanted to get some public data for testing. > that I wished >he would stop sending me mail, that I did not (and do not) want to >hear about his intellectual property, and that his lawyers should be >aware that my document retention policy for email is "forever in a >bank vault" and that I've been known to take formal notes during >telephone calls. Which I obliged after telling you I hate attorneys and I don't have any. If I had an attorney, I wouldn't be allowed to post in the IETF mailing list. Just go read the IP release for posting to the mailing list. > Perhaps Mr. Moore's harping on his "intellectual >property" and an NDA sent me into a raving paranoid break, You were harping on how skeptical you were and I was just trying to convince you I had something important while also saying I could not reveal IP. It would seem to me that if you have public data and you are devoting your life to fighting spam, then you would do as much useful work as you possibly can with that data. But any way, that is your problem, not mine. I've moved on. > but many >of us know real life stories that started similarly and turned out >poorly from some points of view. How many of us get killed in automobile accidents (top 3 killer), yet we still drive. And IP is not deadly. No logical comparison statistically. >He sent a few more messages after my request that he stop, but eventually >quit. I know the technique. Make a bunch of false statements then demand no replies in hopes you have a case in future. I had an obligation to set record straight for my email file before quitting. I told you exactly what I was doing. > If this triggers more, I'll file them with the others, not in >file 13 but in that bank vault. Media's cheap today. Please do. More chance to make sure the record is clear. Too bad you had to do it in public which compels me to respond in public. But I only need this one post. You can post any response and there is no need for me to respond again. All crucial false claims by you in public have been refuted. >I'm writting this only to urge caution. Some people are innocent and >well meaning and only appear otherwise to us paranoids (split personality >and delusions of royal or editorial grandeur as well). Other people >really are dangerous. So are cars. > A few years dealing with spammers or with >patents and intellectual property experts should make anyone spooky. Yes I can imagine that. I will like see "them" come to where I am currently (remember I told you I was temporarily out of the USA), a place where the only law is my relative who will knock them off for mere $100 or a pig and few bottles of beer. :) >I realize the most dangerous people don't seem dangerous. Maybe that >proves Mr. Moore's virtue, or not. Thanks.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> Don't get me wrong, but respectfully, this has *NOTHING* to do with SMTP. >SMTP is not involved and not changed. >> >Therin lies the flaw in your plan, as smtp must continually change in a >distributed fashion in order to effectively reduce the amount of >egregiously time-wasting email that flows through its veins. No you are counting the chickens before the eggs. If you have no more eggs, then you won't get any more chickens. The misuse of SMTP has to be attacked at the root cause, which is my proposal (go study all my posts for full logic), rather than morphing SMTP in a never ending game of "cat and mouse".
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:43 AM 9/7/2003 -0400, you wrote: >On Sun, 07 Sep 2003 13:07:10 +0800, Shelby Moore said: > >> It is a wrong assumption to equate commercial email with bulk email. > >Which is why you're trying to rewrite how bulk email is done in order to deal >with *one segment* of commercial e-mail. Now I understand fully. No I am trying to eliminate (drastically reduce) spam (UBE). And, I am trying to give receivers the ability to opt-in and opt-out of all legitimate bulk email under their own power. In my mind, that is more important than existing legitimate bulk email paradigm. I can understand you and other's vested interest in existing legitimate bulk email paradigm, but I don't agree it is more important. When spam is 99% of all email you recieve, you may begin to care also. It won't be too long from now... Fortunately what this discussion is proving to me so far, is that architectual change will be resisted fiercely by vested groups that control the design of the internet. This means that my new (soon to be patent pending) algorithm for http://AntiViotic.com will have a near monopoly in the market. So for me and users of my service it will be great. But for the rest of users, spammers will be forced to actually increase the amount of email they send in the self-feeding model of my algorithm. Unfortunately, the rest of the email universe will be imploded by an accelerating rate of spam. Just imagine what spammers will do when confronted with an insubvertable algorithm that can detect bulk. I came here to see if there was a better way, but I guess we will have to do it the hard way (and very profitable for me). This is all conjecture (vaporware) at this point...but very well researched conjecture. I'll be back here in this list later (probably a year from now) when your needs have changed to a more dire state regarding email.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:54 AM 9/7/2003 -0400, you wrote: >The evidence indicates that the senders will use whatever is more likely to >result in the receiver seeing the message. This is different from seeing >it where the receiver would like to see it. I get your point and it is a reasonable one that must be taken seriously. However, I think that at some point soon, users will have no choice but to either stop using email (because it doesn't work efficiently any more) or look for a solution to spam. At that point, I think they will demand change (already seeing early warning signs of that now). Also you are assuming that legitimate bulk email is very important to most receivers wherein I think for most spam is a bigger issue. Lastly, I don't think my proposal has to be any more inconvenient than existing sub/unsubscription procedures, which require trading several emails, versus inputting one POP domain in an email client. I understand the concept of diminishing utility in economics, so I take your point seriously. And I will ponder it more. >The point of the usenet reference analogy is that if placing things where >people have to go look was sufficient, folks would not have started using >spam email. But there is a key difference. Users started preferring email without the agreement that it would have spam rate increase from 8% to 50% in 2 years. When the noise gets too high, you turn off the TV and do something more productive (remember antenae reception?). If you have a blackout every 5 minutes, you giveup trying to do anything with appliances. Etc.. > They would have used other techniques. And they will if spam continues to increase. They already are. And those techniques may have much worse effects than a correct architectual definition of spam. For example, legitimate mailing list traffic is getting falsely classified as spam. Maybe not for you, perhaps because you are expert, but I assert your experience is not likely shared equally by the 500 million... > Many (most) of the >spammers are simply interested in any technique that will get the message >in front of the recipient. Hence, they will use the path that is checked >most often (the personal mailbox). Yes my proposal depends on that fact. Once you have the legitimate email separated from the spam architecturally, then you can effectively increase the cost of spamming to the point it is a non-economically viable activity. Remember the economics of spam is precisely what makes it so noisy. If spam ever got like 5% response rate, then it would not be such a big problem as it is. The fact that is averages something like < 0.005% response rate is what makes it both a big noise problem and also makes it very fragile cost wise. The typical individual spammer does not make $300,000 a day. More like less than $100,000 a year. And the institutionalized spammers (the economy-of-scale bundlers) would make nice big fish for FTC. The more you can concentrate, the better. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> > The second is raising the cost to the spammer. Personally, I like the idea of > taking up a collection among the ISPs and other providers, and hiring some good > ethnic muscle (there's competition in the field, a number of experienced and > ruthless groups are available). I'm sure the spam problem would change > drastically if the spammer was seriously having to balance the mentioned $300K > for bogus enhancement pills against having their kneecaps broken by one group > or worse by one of the other groups... What, who needs a kneecap job done? Contact me off-list;) We have very reasonable rates wait. sorry. that was spam.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> From: Shelby Moore <[EMAIL PROTECTED]> > ... >>And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? How did I get involved in this? 15 years ago I had a boss that finally taught me to never use real names in examples or scenarios even when I was sure I was being nice. It's better to invent names, and to take care that they're far from anything real. You must assume that someone will take you literally in bad ways you'll never predict. > ... > Ironically you mention Vernon (is that a hint?), because if I remember > correctly he runs the DCC and recently had a similar unexplained > overtly negative reaction recently when I tried to contact him regarding > some improvements to DCC. I suppose it is possible that your negative > responses here may be explained by similar vested interest with him. > Maybe, but in fairness I won't immediately assume the worst ethics of you. "Worst ethics"? Mr. Moore contacted to ask me to sign a non-disclosure agreement so that I might learn how to make the DCC "effective" based on his new "intellectual property." He was also interested in buying copies of DCC data. My responses started cool and eventually became as "overtly" negative as I could without calling him names. I told him that he had disclosed nothing of his intellectual property to me, that there is no plausible chance we might ever do any business, that I wished he would stop sending me mail, that I did not (and do not) want to hear about his intellectual property, and that his lawyers should be aware that my document retention policy for email is "forever in a bank vault" and that I've been known to take formal notes during telephone calls. Perhaps Mr. Moore's harping on his "intellectual property" and an NDA sent me into a raving paranoid break, but many of us know real life stories that started similarly and turned out poorly from some points of view. He sent a few more messages after my request that he stop, but eventually quit. If this triggers more, I'll file them with the others, not in file 13 but in that bank vault. Media's cheap today. I'm writting this only to urge caution. Some people are innocent and well meaning and only appear otherwise to us paranoids (split personality and delusions of royal or editorial grandeur as well). Other people really are dangerous. A few years dealing with spammers or with patents and intellectual property experts should make anyone spooky. I realize the most dangerous people don't seem dangerous. Maybe that proves Mr. Moore's virtue, or not. > BTW, in case you think my anti-spam business depends on this proposal, > that would be false assumption. Actually one of the key advantages > of my anti-spam algorithm (the automatic white listing of legitimate > bulk email) would be rendered unnecessary if this proposal was put > into effect. I hope I don't need to go read about that automatic white listing and so forth. I find Mr. Moore's technical writing as inpenetrable and painful to read as my own. For now I'll assume that the IETF archives provide sufficient protection. > ... > So you are indeed heavily involved with the DCC! Yes I know that > is a weakness in the way the DCC measures bulk, but that is irrelevant > to the point above. Please notice my resolute failure to ask about that weakness. I've no clue what Mr. Moor's intellecutal property is, except that I suspect it is similar to some ideas related to counting mail from IP addresses mentioned by others in public...and conveniently archived by Google. > If I did not realize it was happening, then I would not have created > http://AntiViotic.com . I'd be using the DCC instead. I also wouldn't > have approached Vernon about improving the DCC (only to be turned away > in nasty tone). That's me, the ol' evil, nasty, non-team playing, selfish, negative, unfair, anti-comerz, kook himself. Another thing that Mr. Moore convinced me during our exchanges is that his notions about the nature of the DCC works differ significantly from mine. There's nothing wrong with that, since without Mr. Moore's intellectual property, the DCC is not "effective" or whatever. I hope the fact that every time I turn around I bump into another person demanding attention for the Ultimate Wonderful Final Perfect Solution to The Spam Problem is a good sign instead of a perverted maturation of the anti-spam industry. Vernon Schryver[EMAIL PROTECTED]
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 7 Sep 2003, Shelby Moore wrote:
>
> >I run a few mail servers, and have built many more. I personally would
> >have no desire for my mail to be handled by POP3, passed in cleartext
> >across the public internet, when I simply log into
> >my machine securely (locally or remotely) and type "mail" to access my
> >email.
>
>
> There is nothing in my proposal that requires your private email to deviate from
> your current more secure methods of SMTP delivery.
>
Shelby, by doing things this way i get no bulk email. Noone in their
right mind would target a server with 15 users for spamming... this
happens to mail providers with millions of subscribers.
> By definition, any thing that is bulk is no longer private, because you can't
> control what another person will do with the same information.
>
>
Why don't we try to eliminate it all in one fell swoop... bulk email, junk
postal mail, telemarketing calls, the whole lot of it. There is an
annoying company called the social security something or other who for
years sent me certified letter after certified letter saying that I should
have one of their numbers, and please contact them.
> > Further, I am not interested in having my mail sit on someone elses
> >server.
>
> Your email would not. Only bulk email which is shared with many other people.
>
Good, because I don't want any bulk mail on my server. You can collect
all you want for me on some random server though. Not interested in
wasting either the cpu cycles or disk space it would take to deal with it.
>
> > Don't get me wrong Shelby, I HAVE a POP3 server, that I built
> >myself, for the use of friends and affiliates. Nor do I have the desire
> >to collect 2-3000 mail boxes in my lifetime, for all of the business
> >relationships I may make during this period.
>
>
> It is a wrong assumption to equate business email with bulk email.
>
---
#!/bin/sh
rant(){
echo (
I am of the opinion that I never want to be sold anything in my life. If
I want it, I can find it.
These people deploying spammers as marketing agents are no different than,
or possibly are, the people who deployed call center sweatshops as
marketing tools, and bulk mailers eating thousands of trees proclaiming
what great deals they have on cars this week. The truth is, if someone
wants to spam bad enough, they will. We could rewrite smtp 5 times, and
it would be engineered, reverse-engineered, reengineered, implemented,
modified, patched, and kludged by thousands of people for various reasons
each time. And every time one of those reasons would be spam. As long as
we live in a profit-driven society anyway. Its too bad we can't just
hold hands and sing cumbyah, but it happens that way sometimes. This has
been rehashed time and time again on this list, unfortunately, which is far
from the venue for this discussion.
Who volunteers his mail server for the spam-discussion-redirect list that
will effectivly be the /dev/null for spam related topics that arrive at
the ietf? For those of you unfamiliar with UNIX or any of its
derivatives, /dev/null is a storage device for important internal system
processes. What say you IETF? Can we engineer a solution to this nasty
problem?
) | /bin/mail -s spam [EMAIL PROTECTED]
sleep 12
rant
}
rant
---
> It is a wrong assumption to equate commercial email with bulk email.
>
rant
>
> > Seems easier than a complete rewrite
> >and redeployment of SMTP, which is what would be necessary inthis
> >instance.
>
>
> Don't get me wrong, but respectfully, this has *NOTHING* to do with SMTP. SMTP is
> not involved and not changed.
>
Therin lies the flaw in your plan, as smtp must continually change in a
distributed fashion in order to effectively reduce the amount of
egregiously time-wasting email that flows through its veins.
> Shelby Moore
> http://AntiViotic.com
>
>
sleekfreak pirate broadcast
world tour 2002-3
live from the pirate hideout
http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
IMO, this (whether Hotmail will implement a specific feature) is a fairly irrelevant (an 80 out of 80/20 rule) fork of the debate relative to the main point of the proposal, so let's try to wrap this fork up with one or two go rounds max okay. >> Interestingly note that Hotmail makes you pay to POP *FROM* hotmail, but no >> charge to POP from other accounts *TO* Hotmail. Does that give you any hint >> about their business model?? > >Yes. It's *NOT* a business model where they want to be polling a dozen servers >on a regular basis for each of their customers for mail that may or may not be >there, and for the average mailing list, probably is not there at any given >poll. Not any more than they want to be POPing any email at all. Nobody wants to do any work they do not have to do. But if there is advantage for them, or a profit to be made, they do it. If they do not, someone else will, then they lose marketshare. They used to not POP email at all (do you remember or did you know that?). Then they discovered they were missing a big market of eyeballs. > They want eyeballs, and the last thing they want to do is expend more >effort than needed to get eyeballs. No disrespect intended, but that sentence is illogical. You want something and you don't want to do something that gives you what you want. You mean I guess that they would not agree to add effort just to retain existing eyeballs. Again I disagree. I think they will do what ever they have to in order to retain market share, as long as the cost doesn't kill more profit than it retains. > Sure - they can even optimize the 'POP the >list' check by only doing it once for all the subscribers - but they're still >hitting each server for each list on a several-times a day basis. And under >the current scheme, they can just *catch* one SMTP transaction with all the >RCPT TO's piggybacked *when there's actual mail*. So they'd have to work a lot >harder under your scheme. POPing once (one list mailing) versus processing one email with zillion RCPT TOs (one list mailing) is not a very big cost difference. One might be slightly less than the other and we really can't say which one, but it is irrelevant because the difference is insignificant. Actually it is more likely that when they POP they will get several messages at once, so less cost than catch several SMTP emails. Also they know a priori the correlation of receivers to POP, which can be optimized with time, versus having to build a new mapping table in real-time every time they process an SMTP with RCPT TO. >And let's *THINK* for a moment here - what is your proposal *REALLY* going to >change? We already have many estimates that 50% or so of all e-mail is spam. >Let's take that as a given, and let's make the rash assumption that the rest is >25% mailing list traffic and 25% person-to-person. It be more interesting to know what the real stats are on the other 50%, because I doubt that 25% is legitimate bulk email. It seems that you live in a different (mailing list centric) world than I and most "normal" people live in. I join mailing lists for a short time to get something done, then I leave asap. Most of the people I know and the many thousands of customers I come into contact with, seem to not even know how to use a mailing list. With 500 million people on the internet, I would venture that 80% don't even know what a mailing list is. They may use Yahoo Personals, and not even realize it is a mailing list. Since the email is being directly deposited into the Yahoo account, they have no clue. Any way, let's follow your line of debate... >So what you want to do is take the 25% of the list traffic that works just fine >on the current infrastructure, No it doesn't work fine. My gf complained that she couldn't find her Yahoo Personals email amongst the 500 spams she gets per day, of course that makes me happy but that is besides the point :) > and is usually quite easily whitelistable via a >number of different methods - Whitelisting can be subverted by spammers: http://www.cnn.com/2003/TECH/internet/09/01/spam.chainletter/index.html "...Herrick, however, admits that the practice could be a good way to bypass e-mail filters which block messages from senders who are not known to the recipient. Spammers could use chain letters to discover the addresses of people with whom you frequently communicate. Spam purporting to be from someone in your address book would sneak by filters. "If I were a spammer, I'd be working very hard to perfect this technique," he said..." > and move it to something totally different. >And what you're left with is a 2-1 mix of spam and personal mail that you >yourself admit things like the DCC and spam filters are unable to perfectly >distinguish. The whole point of the change is to enable elimination of the spam which can not currently be done. See my response to John C Klensin, regarding "chicken and eg
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 13:07:10 +0800, Shelby Moore said: > It is a wrong assumption to equate commercial email with bulk email. Which is why you're trying to rewrite how bulk email is done in order to deal with *one segment* of commercial e-mail. Now I understand fully. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 12:23:19 +0800, Shelby Moore said: > Interestingly note that Hotmail makes you pay to POP *FROM* hotmail, but no > charge to POP from other accounts *TO* Hotmail. Does that give you any hint > about their business model?? Yes. It's *NOT* a business model where they want to be polling a dozen servers on a regular basis for each of their customers for mail that may or may not be there, and for the average mailing list, probably is not there at any given poll. They want eyeballs, and the last thing they want to do is expend more effort than needed to get eyeballs. Sure - they can even optimize the 'POP the list' check by only doing it once for all the subscribers - but they're still hitting each server for each list on a several-times a day basis. And under the current scheme, they can just *catch* one SMTP transaction with all the RCPT TO's piggybacked *when there's actual mail*. So they'd have to work a lot harder under your scheme. And let's *THINK* for a moment here - what is your proposal *REALLY* going to change? We already have many estimates that 50% or so of all e-mail is spam. Let's take that as a given, and let's make the rash assumption that the rest is 25% mailing list traffic and 25% person-to-person. So what you want to do is take the 25% of the list traffic that works just fine on the current infrastructure, and is usually quite easily whitelistable via a number of different methods - and move it to something totally different. And what you're left with is a 2-1 mix of spam and personal mail that you yourself admit things like the DCC and spam filters are unable to perfectly distinguish. To quote Douglas Adams: "Many reasons for this unhappiness were proposed, most of which involved the movement of small green pieces of paper, which was odd given that on the whole, it wasn't the small green pieces of paper that were unhappy..." Having exiled the mailing list traffic, we would then be able to work on separating the spam from non-spam - but as you already noted yourself, we don't know how to do that yet. And getting rid of the mailing list traffic doesn't in fact gain us anything at all, since everybody who filters list traffic into separate folders for each list knows that isn't the problem - it's the unfiltered stuff that's left in the inbox. I'll note in passing that the two highest SpamAssassin scores I've ever seen were both on legitimate postings to mailing lists - both were humor pieces about spam Quite frankly, given that at least half the spam I get is already in obvious violation of at least one law (pick one - securities fraud, advance-fee scams, wire fraud, bogus pharmeceuticals, or hijacking a proxy to send the mail), I severely doubt that anything the IETF does in regards to standards won't make a difference. The spammers often don't even bother following RFC822 - why should they follow your scheme? The *only* two ways to get rid of spam both involve making it non-profitable. The first is lowering the generated income. Given that recently, somebody hacked the site of a "fertilizer for your body part" scam, and found a list of 6,000 people who had paid $50 a bottle, I have to sadly conclude that Korbluth and Barnum were both correct, there's one born every minute and the rate is increasing. So there's no joy to be found there. The second is raising the cost to the spammer. Personally, I like the idea of taking up a collection among the ISPs and other providers, and hiring some good ethnic muscle (there's competition in the field, a number of experienced and ruthless groups are available). I'm sure the spam problem would change drastically if the spammer was seriously having to balance the mentioned $300K for bogus enhancement pills against having their kneecaps broken by one group or worse by one of the other groups... Pity that will never work though. At least not officially (although one infamous New Zealander apparently retired recently...) pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>I run a few mail servers, and have built many more. I personally would >have no desire for my mail to be handled by POP3, passed in cleartext >across the public internet, when I simply log into >my machine securely (locally or remotely) and type "mail" to access my >email. There is nothing in my proposal that requires your private email to deviate from your current more secure methods of SMTP delivery. By definition, any thing that is bulk is no longer private, because you can't control what another person will do with the same information. > Further, I am not interested in having my mail sit on someone elses >server. Your email would not. Only bulk email which is shared with many other people. > Don't get me wrong Shelby, I HAVE a POP3 server, that I built >myself, for the use of friends and affiliates. Nor do I have the desire >to collect 2-3000 mail boxes in my lifetime, for all of the business >relationships I may make during this period. It is a wrong assumption to equate business email with bulk email. >IF you want to get rid of spam, try communism, wherein there is no need >for commercial messages of any type. It is a wrong assumption to equate commercial email with bulk email. > Seems easier than a complete rewrite >and redeployment of SMTP, which is what would be necessary inthis >instance. Don't get me wrong, but respectfully, this has *NOTHING* to do with SMTP. SMTP is not involved and not changed. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Valdis has identified some of the technical issues associated with using >POP3 in this way. I have refuted all of Valdis's technical points so far. > Let me step back and look at your proposal from another >angle. Yes I think that is productive to discuss the end game (outside of technical issues). >>From the standpoint of the bulk mailer/ poster of material, there is no >advantage (and some disadvantages) of doing that posting so that interested >users can "pull" it relative to just posting the material on a web page >somewhere. Functionally, what you have proposed seems, to me, to be >roughly equivalent to: > > * distributors of bulk materials are required to post them on > selected web sites. It is an acceptable analogy, except I disagree that bulk posters of material favor web post over POP post. I think they favor what ever their receiver favors, by the law of supply and demand and economics. If receivers find it more convenient to get a copy in their InBox, then that will be accomodated (just as with mailing lists and archives now). But in terms of the analogy, I will follow along... > * non-bulk materials may continue to go out via conventional email. Agreed. >Nice plan. The problem is that spammers won't play Agreed. > and efforts to coerce >them into playing will largely fail due to international issues, lack of >adequate incentives, etc exactly the same problem we have today with >state laws prohibiting spam. Here is where I *strongly* disagree. Let me start with a story. The genesis for this proposal came from the fact that our outgoing business email (not bulk but single emails sending a password to a customer, etc) is being blacklisted by SPEWS (etc) because Rackspace (our Host) allowed some other customers of theirs to send spam on the same C class IP range as ours. SPEWS then blacklists the entire C class. Well SPEWS is uncorrectable lately because they've been under DoS attacks (from spammers presumably), thus caches of blacklists are used and nothing can be done except for us to change our email IP address. So in discussing this with the AUP manager and her boss at Rackspace, it became clear that Rackspace would never be able to guarantee the quality of a C class range of IPs, because "Rackspace can not determine what is legitimate bulk email and what is spam and thus can not terminate new customers until a very heavy proof of spamming has already occured, by which time the damage to C class has already been done". So the lesson learned was that if Rackspace could automatically detect high quantities of bulk email in real-time, then with my proposed architectual change, Rackspace could in real-time shut off the spammer. Okay so that is one example of how the architectual paradigm changes the rules and allows more effective actions against spammers. Now take for example legislative combined with ISP. For right now, spammers are avoiding open relays and many foreign IPs because of blacklists, so they get a dialup account and send from there. Well if there was a law requiring USA ISPs to detect and shut these off in real-time, then spammers would need to revert back to open relays and foreign IPs which are effectively dealt with using blacklists. Then blacklists would not have to be so draconian with IP C ranges in countries with strict enforcement, which would make the blacklists more effective and useable. Then take anti-spam software like the DCC, BrightMail, or even our AntiViotic.com. If we know all bulk is bad, the game gets simpler because no whitelist needed. Since whitelists are data that is forgeable by spammer, this closes another hole. No to mention that whitelists make current anti-spam less useable and realistic on wide scale. I could go on... but I hope you begin to see how everything to fight spam depends circularly on the ability to architectually define it. If you can't measure it, you can't do any thing about it. That is a fundamental datum of science. >More generally, you have just defined an "opt in" model that assumes that >anyone who has not explicited opted to receive particular messages will be >able to get them (or be sent them) only be some overt action on the >would-be recipient's part. That is the definition of opt-in. > We know from experience that such a model won't >work without significant legal pressure and enforcement -- if you don't >believe me, sample any reasonable quantity of spam for messages that claim, >quite strongly, that, if you hadn't opted in, you wouldn't be receiving it. What you are saying IMO, is that you can't force bulk emailers or spammers to use opt-in. That has been because you can't measure the spam (UBE) from the legitimate. It is a chicken and egg problem. Once you have the egg (the architectural metric), then reasonable to make the chicken. So comparing to before you had the egg, is not necessari
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Hello Shelby, I run a few mail servers, and have built many more. I personally would have no desire for my mail to be handled by POP3, passed in cleartext across the public internet, when I simply log into my machine securely (locally or remotely) and type "mail" to access my email. Further, I am not interested in having my mail sit on someone elses server. Don't get me wrong Shelby, I HAVE a POP3 server, that I built myself, for the use of friends and affiliates. Nor do I have the desire to collect 2-3000 mail boxes in my lifetime, for all of the business relationships I may make during this period. IF you want to get rid of spam, try communism, wherein there is no need for commercial messages of any type. Seems easier than a complete rewrite and redeployment of SMTP, which is what would be necessary inthis instance. Scott
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
> I merely pointed out that in all >probability, >you haven't actually *tried* doing what you suggest, because it's not anywhere >near as usable as you might think. How could you know if does not yet exist? We can discuss facts and theories of how it would work. But no one can actually test how it does work, because it does not exist yet. The comparison of setting POP accounts in an email client and popping the email, versus subscribing to that many bulk email lists and popping the email, is not useful because no email client has yet been designed to optimize popping from many anonymous POP accounts. >If I'm involved with the DCC in any way shape or form other than the fact I've >seen Vernon write about it on IETF lists, it's such a covert involvment that >even I am unaware of it. Ok cool. Let's move on. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> Hotmail and Yahoo already support and encourage the pulling of email from POP >> accounts. You don't have to enter the POP accounts every time, just once. > >Hmm.. so Hotmail is willing to maintain the list of 40 or 50 places it has >to POP your >mail from for you? I can not predict what Hotmail will do. But given they have an incentive to get you to come their web site as often as possible, and given it is the same bandwidth cost for them, with the potential of making their service more valuable, then my guess is they would do it. Interestingly note that Hotmail makes you pay to POP *FROM* hotmail, but no charge to POP from other accounts *TO* Hotmail. Does that give you any hint about their business model?? > Or are you re-entering it every time you use a new kiosk >someplace? If they are going to offer the feature, then no reason for them not to persist the configuration. They would have no incentive to offer the feature and make it impossible to use efficiently. >(Remember that the major business case for webmail is that you have a *VERY* >small amount of state carried with you (a userid/password pair - >specifically so you >can check mail with a minimum of local resources). Now you are supporting my case above.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 11:53 PM 9/6/2003 -0400, you wrote: >Actually, the point is that there was no way, even within usenet, to >prevent pollution of individual groups with innappropriate spam or off >topic messages. Many groups have fallen into disuse for this reason. Afaics, that is irrelevant, because even a mailing list can be polluted with spam. Your point is against the nature of a public list or group, but my proposal is not designed to fix that problem. My proposal is designed to fix the problem of receivers being forced to receive bulk email from any sender. My proposal forces email addresses to not be public to legitimate bulk email. And then all other bulk senders can be dealt with more effectively. Then of course this does solve the problem for mailing lists also in the end, because then all bulk email gets killed (by a combination of legislative, judicial, ISP, Host, and software features). The fundamental problem with doing any thing about spam, is there is no way to measure it, because the subjective measurement of "unsolicited" is not architectually defined. The point is that if I choose to receive email from list or group (irregardless of whether that list or group has effective policies to prevent incoming spam posts), then it would be helpful to *architectually* differentiate that which I subjectively decide is legitimate bulk email senders from that which I decide is UBE (spam) senders. Once you define the architecture, then all kinds of important features can be built on top of it to deal more effectively with spam ranging from legislative, judicial, ISP, Host, and software. >The other point to understand is that the paradigms (usenet vs email) are >and have been different for very good reasons. And if one assumes that >both paradigms are desired, then it is necessary to address how one >enforces the separation. No one to date has figured out even a set of >definitions to address the needs, much less mechanisms. I am unclear how you think this applies to my proposal? Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
--On Saturday, September 06, 2003 8:22 PM +0800 Shelby Moore <[EMAIL PROTECTED]> wrote: Request for opinions on whether to creating a working group or publish the following idea as an internet draft? Spam is big problem that is getting worse. BrightMail.com (which claims to process 10% of world's email) claims that the percentage of spam out of all email has grown from 16% in Jan. 2002 to 50% in Aug. 2003. A fundamental unsolved problem of doing any thing about spam, is there is currently no unambiguous definition of spam as an enforceable internet standard. This has been architectually impossible to define because the receiver is the subjective determinant of which bulk email is solicited and which is spam (UBE). ... Shelby, Valdis has identified some of the technical issues associated with using POP3 in this way. Let me step back and look at your proposal from another angle. From the standpoint of the bulk mailer/ poster of material, there is no advantage (and some disadvantages) of doing that posting so that interested users can "pull" it relative to just posting the material on a web page somewhere. Functionally, what you have proposed seems, to me, to be roughly equivalent to: * distributors of bulk materials are required to post them on selected web sites. * non-bulk materials may continue to go out via conventional email. Nice plan. The problem is that spammers won't play and efforts to coerce them into playing will largely fail due to international issues, lack of adequate incentives, etc exactly the same problem we have today with state laws prohibiting spam. More generally, you have just defined an "opt in" model that assumes that anyone who has not explicited opted to receive particular messages will be able to get them (or be sent them) only be some overt action on the would-be recipient's part. We know from experience that such a model won't work without significant legal pressure and enforcement -- if you don't believe me, sample any reasonable quantity of spam for messages that claim, quite strongly, that, if you hadn't opted in, you wouldn't be receiving it. Sorry, but no cigar. john
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 11:38:19 +0800, Shelby Moore said: > Please stop the personalized attacks and stick to the facts. I didn't make a personalized attack. I merely pointed out that in all probability, you haven't actually *tried* doing what you suggest, because it's not anywhere near as usable as you might think. ... and then... > So you are indeed heavily involved with the DCC! Yes I know that is a > weakness in the way the DCC measures bulk, but that is irrelevant to the point > above. If I'm involved with the DCC in any way shape or form other than the fact I've seen Vernon write about it on IETF lists, it's such a covert involvment that even I am unaware of it. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 11:51:19 +0800, Shelby Moore said: > Hotmail and Yahoo already support and encourage the pulling of email from POP > accounts. You don't have to enter the POP accounts every time, just once. Hmm.. so Hotmail is willing to maintain the list of 40 or 50 places it has to POP your mail from for you? Or are you re-entering it every time you use a new kiosk someplace? (Remember that the major business case for webmail is that you have a *VERY* small amount of state carried with you (a userid/password pair - specifically so you can check mail with a minimum of local resources). pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>Another reason why you need unique userid/logins for each subscriber - so that >you can prevent forging a UIDL for somebody else to keep them from reading the >message. Being able to do this (and if you have a shared userid, it's almost >impossible to prevent) would make the Bernstein/Bush flamefest regarding list >censorship look trivial in comparison... Again I will repeat: You apparently did not understand the concept. The per user UIDL tracking is stored in the email client, not in the POP server. >Oh... there's also this thing called "webmail". Lots of people use them so >they can >get mail no matter where they are. You can get mail no matter where you are with a POP account also. Most people use webmail either because it is free, or because it is a convenient interface on their POP account. > Lots of these people will be >fundamentally stuck >under your proposal, because every time they used a kiosk they'd have to enter >in all 20-30-40 POP servers they had mailing lists on. Again cars today have rubber tires and titanium rims, not wooden wheels. Applications can improve. It actually happens most of the time. Hotmail and Yahoo already support and encourage the pulling of email from POP accounts. You don't have to enter the POP accounts every time, just once. > Surprisingly enough, >hotmail.com and yahoo.com have a lot of subscribers, you might want to factor >that into your plan How many more silly personal attacks?
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
>> . It is quite a low bandwidth operation (probably less than 1K bytes) to >poll >> a POP server for email > >It's not the bandwidth - it's the fact that there are these annoying things >called >"timeouts". For *each* server that isn't reachable, you get to wait a >minute or >so - suddenly those 150 checks are taking quite some time. And the queuing >theory >of 150 sites with intermittent connectivity doing a push to your POP server is >different than what you see when you try to do 150 pulls First, you are using a baseline of 150 in your example, which is attuned perhaps to your power use given you stated that you manage 6,000 mailing lists. I doubt the average user would subscribe to more than 10 on a regular basis. Second, there is a way of dealing with timeouts on the client side, it is called multithreading. Multithreading (for those readers who might not know, if any) is a fundamental programming concept that all internet clients (and all computer programs and operating systems) depend on. Thus the culmulative delay is not O(n), it is more like Max(n). Thirdly, with your queuing point, you are making the mistake of assuming that the majority of email received is *legitimate* bulk email. I assert this is far from the case for the majority of people. Any way, for the power cases like yours, what is the big deal with setting your email client to run for a few minutes each day, while you go browse some research at google.com or fill your coffee cup? Is your personal paradigm of immediacy for legitimate bulk email a good priority for the internet as a whole, when the very nature of email itself is being doomed by spam? >But of course, if you actually *tried* this, you'd understand it... Please stop the personalized attacks and stick to the facts. >> the same messages over and over again. This would either require a >special modification >> to the POP server and require each user to login with a unique user name. > >Hey.. what did I tell you? Everybody needs a login of their own... Er...you conveniently ignored the "or" that followed the "either": -I wrote (and you ignored)- Or better, users' email clients can be made smarter because there is a UIDL command in both POP3 and IMAP4. This unique identifier can be used by the email client to only download messages which are new to that user. One would assume that POP servers could also remove messages older than say 1 month or so (configurable by the administrator). >> And as a side benefit, there would be no way for someone to subscribe me to a >> list without my permission, as can be done by sniffing an authentication >email >> for Majordomo. > >If your confirmation mail is being sniffed, you have *BIGGER* issues. Again you conveniently ignored the overall point which followed. -I wrote (and you ignored)- And no way for someone to subscribe me to a list that has no public instructions for subscribing or unsubscribing (i.e. spam in guise of business email). That seems to be a BIG problem that many people have these days, probably even yourself. >> The flags can either be stored >at the POP server >> and then give each user a unique login id, > >Hey.. there's that unique login again. Er...you conveniently ignored the "or" that followed the "either": -I wrote (and you ignored)- , or more realistically just let email clients manage their own flags using UIDL. >> No only 6000 POP accounts. See above how email clients can handle the >> detection of new messages using UIDL. And you only need one anonymous login >> and no password (just configure the POP server to accept any login and >> password). > >And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? Yes your email client can keep track of which UIDL's it has already downloaded. Keith's email client can keep track of his. Vernon his. Ironically you mention Vernon (is that a hint?), because if I remember correctly he runs the DCC and recently had a similar unexplained overtly negative reaction recently when I tried to contact him regarding some improvements to DCC. I suppose it is possible that your negative responses here may be explained by similar vested interest with him. Maybe, but in fairness I won't immediately assume the worst ethics of you. BTW, in case you think my anti-spam business depends on this proposal, that would be false assumption. Actually one of the key advantages of my anti-spam algorithm (the automatic white listing of legitimate bulk email) would be rendered unnecessary if this proposal was put into effect. >> >Have you actually *TRIED* to use more than 100 POP accounts under any >current >> >mail software? > >> I will respond with similarly rhetorical question. Did you try to use >> Netscape 2 on most current web pages? Why make any application RFC if there >> can be no progress in applications? > >There
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sat, 06 Sep 2003 23:07:44 EDT, [EMAIL PROTECTED] said: > And as I pointed out, you'll need to create 30,000, because one account doesn't > allow you to keep track of who has already seen what messages. And no, you're > *NOT* allowed to just say "everybody can fetch all the UIDLs and we'll just tag > them with the subscriber ID" - go read and *UNDERSTAND* section 6.2 of RFC2298 > in order to understand why. Another reason why you need unique userid/logins for each subscriber - so that you can prevent forging a UIDL for somebody else to keep them from reading the message. Being able to do this (and if you have a shared userid, it's almost impossible to prevent) would make the Bernstein/Bush flamefest regarding list censorship look trivial in comparison... Oh... there's also this thing called "webmail". Lots of people use them so they can get mail no matter where they are. Lots of these people will be fundamentally stuck under your proposal, because every time they used a kiosk they'd have to enter in all 20-30-40 POP servers they had mailing lists on. Surprisingly enough, hotmail.com and yahoo.com have a lot of subscribers, you might want to factor that into your plan pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 09:58:47 +0800, Shelby Moore said: > If it became an RFC or internet standard, and it became widely adopted, then >it is reasonable to assume that email clients would add features to handle this > . It is quite a low bandwidth operation (probably less than 1K bytes) to poll > a POP server for email It's not the bandwidth - it's the fact that there are these annoying things called "timeouts". For *each* server that isn't reachable, you get to wait a minute or so - suddenly those 150 checks are taking quite some time. And the queuing theory of 150 sites with intermittent connectivity doing a push to your POP server is different than what you see when you try to do 150 pulls But of course, if you actually *tried* this, you'd understand it... > However, there is one key technological hurdle I did miss in my haste, there > would need to be some mechanism so that the same user doesn't keep downloading > the same messages over and over again. This would either require a special > modification > to the POP server and require each user to login with a unique user name. Hey.. what did I tell you? Everybody needs a login of their own... > And as a side benefit, there would be no way for someone to subscribe me to a > list without my permission, as can be done by sniffing an authentication email > for Majordomo. If your confirmation mail is being sniffed, you have *BIGGER* issues. And if you have bigger issues, I suggest using the *proper* tools for the job. See RFCs 2362-2364 and 3156. If your issues are bigger than that, e-mail subscriptions are the least of your problems. > False. You are correct that I missed this issue in my initial post. > However, it need be only one POP account (one storage of emails) with flags for > each user. In other words, the storage requirement need no increase > drastically with number of subscribed users. Hmm... store each mail as an object with links for each recipient. A truly novel idea, our homegrown mail system implemented it back in 1992. > The flags can either be stored at the > POP server > and then give each user a unique login id, Hey.. there's that unique login again. > No only 6000 POP accounts. See above how email clients can handle the > detection of new messages using UIDL. And you only need one anonymous login > and no password (just configure the POP server to accept any login and > password). And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? > >Have you actually *TRIED* to use more than 100 POP accounts under any current > >mail software? > I will respond with similarly rhetorical question. Did you try to use > Netscape 2 on most current web pages? Why make any application RFC if there > can be no progress in applications? There's a difference - I'm not proposing a new scheme of doing things that involves a change to 500 million users. So I submit to you that if this idea is too hard to use with the current version of Outlook, it is a *non starter* as a practical matter. > 50, even 5000, is not statistically bulk on internet scale. Is it not > possible (or likely) to write laws without exclusions? Do you think Hosts, > ISPs, and anti-spam software would not account for this statistical phenomenon? Only problem is that many spammers are *already* only dropping 40-50 copies of a note at a site at a time, specifically to work around that - then the rest of the spamming recipients at your site get a different version with a different From: line and a different source IP address. I submit to you that if you didn't realize this was happening, you may not be qualified to be suggesting proposals to counter it (Hint - if spammers weren't doing this, it would be trivial to block them, and we'd not be HAVING this discussion, right? ;) > >It's ironic that you're proposing this on a push-based mailing list provided by > >an organization that is probably not in a position to provide POP accounts for > >the 30,000 or so recipients of the the list. > No. As I said above, they would only need to provide one POP account for this > mailing list. And as I pointed out, you'll need to create 30,000, because one account doesn't allow you to keep track of who has already seen what messages. And no, you're *NOT* allowed to just say "everybody can fetch all the UIDLs and we'll just tag them with the subscriber ID" - go read and *UNDERSTAND* section 6.2 of RFC2298 in order to understand why. You might also want to go re-read the ASRG mailing list archives, your proposal (and variants thereof) has been kicked down the beach like a dead whale multiple times already. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
One additional point in response to this point: >So let's see.. Currently, if your bank sells your e-mail address to another >company, >you get spammed. So instead, you'll have it so that you check your bank's POP >server in case there's important mail about your mortgage. Seems like the >obvious >scheme is for the bank to charge the other company to put stuff in your POP >mailbox. > >So you still get spammed... In addition to what I wrote below, remember that when the bank is sending me an individual email regarding some business I am conducting with them, they are not sending that email in bulk (to any one else). They could send that directly to my email. If they are indeed sending a similar email to ALL their clients at the same time, then in that case they are sending bulk email. So with my proposal, it just forces business to separate their business email from their marketing bulk email. If I trust my bank to send only important bulk email, then I can add that POP account to ones that my email client checks regularly (as regularly as I chose, not as my bank choses...gives me the control). Given that email is insecure transmission medium, no business should be sending me anything too important in email. They had better have an alternative means of getting in contact with me about important and urgent matters. ---I wrote before only No. Because you can chose to not check it and/or you at least know who is spamming you and can hold them responsible directly. Thus your bank would stop doing it, because they make $ by not losing your business.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Thanks for the feedback and giving me a chance to clarify some issues. >This is broken in two distinct ways: Disagree. Read on. >1) I as a mail user now have to go check 150 POP servers several times a day >for all the various lists I'm on - many of the lists are low-volume, but I'd >have >to go CHECK every day just in case something DID get posted. If it became an RFC or internet standard, and it became widely adopted, then it is reasonable to assume that email clients would add features to handle this. It is quite a low bandwidth operation (probably less than 1K bytes) to poll a POP server for email. I assume it would become popular because as shown with further logic in my responses below, the idea provides strong benefits to all (except spammers). However, there is one key technological hurdle I did miss in my haste, there would need to be some mechanism so that the same user doesn't keep downloading the same messages over and over again. This would either require a special modification to the POP server and require each user to login with a unique user name. Or better, users' email clients can be made smarter because there is a UIDL command in both POP3 and IMAP4. This unique identifier can be used by the email client to only download messages which are new to that user. One would assume that POP servers could also remove messages older than say 1 month or so (configurable by the administrator). And as a side benefit, there would be no way for someone to subscribe me to a list without my permission, as can be done by sniffing an authentication email for Majordomo. And no way for someone to subscribe me to a list that has no public instructions for subscribing or unsubscribing (i.e. spam in guise of business email). >> In the case of a public distribution (e.g. most direct email and mailing >> lists), a POP3 (and IMAP) account of user "anonymous" with password "none" >> would suffice. In the case of private dissemination (private mailing >lists), a >> POP3 (and IMAP) server with individual accounts could be provided. > >Nope. even for a public list you get to keep a separate POP3 account for each >subscriber - if one person has checked for postings yesterday, but another >hasn't >since last Tuesday, you can't feed the right list to each person. False. You are correct that I missed this issue in my initial post. However, it need be only one POP account (one storage of emails) with flags for each user. In other words, the storage requirement need no increase drastically with number of subscribed users. The flags can either be stored at the POP server and then give each user a unique login id, or more realistically just let email clients manage their own flags using UIDL. >2) I as the administrator of a site that hosts 6,000 mailing lists just got the >additional aggrivation of providing POP3 service for 700,000 e-mail addresses >(yes, we've got that many). This includes "My password doesn't work" support >and things like that. Gee thanks. No only 6000 POP accounts. See above how email clients can handle the detection of new messages using UIDL. And you only need one anonymous login and no password (just configure the POP server to accept any login and password). Now instead of sending 700,000 emails for each email sent to all your 6000 of you lists, you only send 6000 emails or less. Now instead of managing bounces, keeping your IP off of blacklists, hassling with subscribing and unsubscribing the users, then all you have to do is publish the domain names of your 6,000 POP servers on a web page. The flow of noise is probably greatly reduced. >Have you actually *TRIED* to use more than 100 POP accounts under any current >mail software? I will respond with similarly rhetorical question. Did you try to use Netscape 2 on most current web pages? Why make any application RFC if there can be no progress in applications? >> 1. Any bulk email is then spam (receiver has not opted in) and can be dealt >> with by ISPs, Hosts, legislators, judiciaries, and anti-spam software. > >So I drop a note to 50 friends inviting them to a barbecue, and I end up in >the slammer. 50, even 5000, is not statistically bulk on internet scale. Is it not possible (or likely) to write laws without exclusions? Do you think Hosts, ISPs, and anti-spam software would not account for this statistical phenomenon? >> 2. Receivers now have uniform control over opt-in/opt-out policy without a >global authority > >This actually means "We've pushed the headache to the recipients". How so? In my mind, I find it to be no more of a headache than subscribing and unsubscribing to a mailing list. And certainly a lot less of a headache than trying to opt-out of a list that won't let you opt-out. I think recipients already have a big headache, it is called "spam". And it is getting worse. It is predicted that very soon 90% of all email sent will be spam. When t
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sat, 06 Sep 2003 20:22:03 +0800, Shelby Moore <[EMAIL PROTECTED]> said: > Simply define that legitimate bulk distribution of email should be done by > mechanism of each bulk distributor providing a public POP3 (and IMAP) account > or server, rather than sending the email directly. This is broken in two distinct ways: 1) I as a mail user now have to go check 150 POP servers several times a day for all the various lists I'm on - many of the lists are low-volume, but I'd have to go CHECK every day just in case something DID get posted. > In the case of a public distribution (e.g. most direct email and mailing > lists), a POP3 (and IMAP) account of user "anonymous" with password "none" > would suffice. In the case of private dissemination (private mailing lists), a > POP3 (and IMAP) server with individual accounts could be provided. Nope. even for a public list you get to keep a separate POP3 account for each subscriber - if one person has checked for postings yesterday, but another hasn't since last Tuesday, you can't feed the right list to each person. So that brings us to: 2) I as the administrator of a site that hosts 6,000 mailing lists just got the additional aggrivation of providing POP3 service for 700,000 e-mail addresses (yes, we've got that many). This includes "My password doesn't work" support and things like that. Gee thanks. > The elegance of this paradigm is that users then control the opt-in/opt-out > database, by configuring their email client to POP email from only the bulk POP > accounts they wish to subscribe to. Have you actually *TRIED* to use more than 100 POP accounts under any current mail software? > 1. Any bulk email is then spam (receiver has not opted in) and can be dealt > with by ISPs, Hosts, legislators, judiciaries, and anti-spam software. So I drop a note to 50 friends inviting them to a barbecue, and I end up in the slammer. > 2. Receivers now have uniform control over opt-in/opt-out policy without a global > authority This actually means "We've pushed the headache to the recipients". > 3. Legitimate bulk senders can be insured that they or their email won't be > misclassified as spam So.. you ready to have every single eBay or Amazon customer check their POP account there every day just in case there's important mail for them? > 4. Those who send UBE can no longer claim they are legitimate or that > receiver has opted-in (ambiguity removed) and can be dealt with by ISPs, Hosts, > legislators, judiciaries, and anti-spam software. Well.. maybe. But.. > 5. With a "pull" paradigm, the load (resource usage) on the public internet, > sender, and receiver is reduced, because I venture that a majority of bulk emai > l sent would not be pulled. So let's see.. Currently, if your bank sells your e-mail address to another company, you get spammed. So instead, you'll have it so that you check your bank's POP server in case there's important mail about your mortgage. Seems like the obvious scheme is for the bank to charge the other company to put stuff in your POP mailbox. So you still get spammed... > their hands would not longer be bound by ambiquity. I realize that some vested > interests, such as direct emailers or those invested in push based mailing > lists, might resist. It's ironic that you're proposing this on a push-based mailing list provided by an organization that is probably not in a position to provide POP accounts for the 30,000 or so recipients of the the list. Baby with the bathwater, Shelby... Baby with the bathwater. pgp0.pgp Description: PGP signature
Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Request for opinions on whether to creating a working group or publish the following
idea as an internet draft?
Spam is big problem that is getting worse. BrightMail.com (which claims to process
10% of world's email) claims that the percentage of spam out of all email has grown
from 16% in Jan. 2002 to 50% in Aug. 2003.
A fundamental unsolved problem of doing any thing about spam, is there is currently no
unambiguous definition of spam as an enforceable internet standard. This has been
architectually impossible to define because the receiver is the subjective determinant
of which bulk email is solicited and which is spam (UBE).
ISPs, Hosts, legislators, judiciaries, and even anti-spam software, have a fundamental
problem in that definition of spam as UBE is currently architectually unenforceble due
the fact that subjective determination of "unsolicited" current happens after the
email has been delivered to the receiver.
My idea is to create an internet draft, RFC, and hopefully internet standard, that
would define a simple architectual paradigm for legitimate bulk email that
unambiguously separates it from spam (UBE).
Simply define that legitimate bulk distribution of email should be done by mechanism
of each bulk distributor providing a public POP3 (and IMAP) account or server, rather
than sending the email directly.
In the case of a public distribution (e.g. most direct email and mailing lists), a
POP3 (and IMAP) account of user "anonymous" with password "none" would suffice. In
the case of private dissemination (private mailing lists), a POP3 (and IMAP) server
with individual accounts could be provided.
The elegance of this paradigm is that users then control the opt-in/opt-out database,
by configuring their email client to POP email from only the bulk POP accounts they
wish to subscribe to.
The effort to support this paradigm is minimal because it uses existing email
paradigm. Legitimate bulk senders have to change from a broadcast ("push") metaphor
(e.g. Majordomo) to a "pull" metaphor simply by depositing their outgoing email in the
public POP account they create. Receivers simply follow instructions to POP bulk
email they want, instead of the equally complex task of subscribing to bulk email.
This accomplishes several goals:
1. Any bulk email is then spam (receiver has not opted in) and can be dealt with by
ISPs, Hosts, legislators, judiciaries, and anti-spam software.
2. Receivers now have uniform control over opt-in/opt-out policy without a global
authority
3. Legitimate bulk senders can be insured that they or their email won't be
misclassified as spam
4. Those who send UBE can no longer claim they are legitimate or that receiver has
opted-in (ambiguity removed) and can be dealt with by ISPs, Hosts, legislators,
judiciaries, and anti-spam software.
5. With a "pull" paradigm, the load (resource usage) on the public internet, sender,
and receiver is reduced, because I venture that a majority of bulk email sent would
not be pulled.
I think this paradigm would empower Hosts, ISPs, legislatures, and judiciaries to do
more about spam (incoming) and spammers (outgoing), because their hands would not
longer be bound by ambiquity. I realize that some vested interests, such as direct
emailers or those invested in push based mailing lists, might resist. However, I
think the benefits outweigh the limited costs to migrate. Some direct emailers might
resist because some may prefer being able to cloak spam under the guise of
"solicited". Legitimate bulk emailers stand to gain a lot by separating themselves
from the noise of UBE.
Shelby Moore
http://AntiViotic.com
