Paragraph Length Limit (new rule)
Hello, We received a new type of spam, twice, and we are not willing to give them a third chance. The body includes a long html paragraph (...) of headlines from the news. The following works at the command line: perl -p0e 's/((?:(?!<\/p>).){999,}<\/p>)/-->$1<--/msig' example.eml perl -n0e '/((?:(?!<\/p>).){999,}<\/p>)/msig and print "--->$1<---"' example.eml The following SA rule, however, does not work at all: rawbody __B_PLL /(?:(?!<\/p>).){999,}<\/p>/msi tflags __B_PLL multiple maxhits=1 meta B_PLL __B_PLL describe B_PLL Body: Paragraph Length Limit score B_PLL 1.0 I would be most grateful if you could spot the but in the above rule.
Re: Paragraph Length Limit (new rule)
>I would be most grateful if you could spot the but in the above rule. The *bug*, sorry.
Re: Paragraph Length Limit (new rule)
On 08/03/2016 12:07 PM, Ruga wrote: Hello, We received a new type of spam, twice, and we are not willing to give them a third chance. The body includes a long html paragraph (...) of headlines from the news. The following works at the command line: perl -p0e 's/((?:(?!<\/p>).){999,}<\/p>)/-->$1<--/msig' example.eml perl -n0e '/((?:(?!<\/p>).){999,}<\/p>)/msig and print "--->$1<---"' example.eml The following SA rule, however, does not work at all: rawbody __B_PLL /(?:(?!<\/p>).){999,}<\/p>/msi tflags __B_PLL multiple maxhits=1 meta B_PLL __B_PLL describe B_PLL Body: Paragraph Length Limit score B_PLL 1.0 I would be most grateful if you could spot the but in the above rule. please pastebin a sample msg
Re: Paragraph Length Limit (new rule)
echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 999 | head -n 1 )" >example.txt spamassassin -t -D B_LLL.rule wrote please pastebin a sample msg
Re: Paragraph Length Limit (new rule)
Am 03.08.2016 um 12:49 schrieb Ruga: echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 999 | head -n 1 )" >example.txt spamassassin -t -D B_LLL.rule you where asked for a real *mail* example instead some generic stuff On Wed, Aug 3, 2016 at 12:15 PM, Axb <'axb.li...@gmail.com'> wrote please pastebin a sample msg signature.asc Description: OpenPGP digital signature
Re: Paragraph Length Limit (new rule)
I am AWAY for my office. Real spam truly unnecessary. Sent from ProtonMail Mobile On Wed, Aug 3, 2016 at 12:51 PM, Reindl Harald <'h.rei...@thelounge.net'> wrote: Am 03.08.2016 um 12:49 schrieb Ruga: > echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold > -w 999 | head -n 1 )" >example.txt > > spamassassin -t -D B_LLL.rule On Wed, Aug 3, 2016 at 12:15 PM, Axb <'axb.li...@gmail.com'> wrote >> please pastebin a sample msg
Re: Paragraph Length Limit (new rule)
Keep in mind we do not know that. It is better to not reply and wait a few hours than get Reindl worked up. :) > On Aug 3, 2016, at 5:55 AM, Ruga wrote: > > I am AWAY for my office. > Real spam truly unnecessary. > > Sent from ProtonMail Mobile > > > On Wed, Aug 3, 2016 at 12:51 PM, Reindl Harald <'h.rei...@thelounge.net'> > wrote: >> >> Am 03.08.2016 um 12:49 schrieb Ruga: >> > echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold >> > -w 999 | head -n 1 )" >example.txt >> > >> > spamassassin -t -D B_LLL.rule > >> you where asked for a real *mail* example instead some generic stuff >> >> > On Wed, Aug 3, 2016 at 12:15 PM, Axb <'axb.li...@gmail.com'> wrote >> >> please pastebin a sample msg >>
Re: Paragraph Length Limit (new rule)
Am 03.08.2016 um 12:55 schrieb Ruga: I am AWAY for my office then reply when you are back Real spam truly unnecessary says who in case of writing a sane rule? to understand the context it is necessary and in any case prefered to dig in the dark with some isolated stuff On Wed, Aug 3, 2016 at 12:51 PM, Reindl Harald <'h.rei...@thelounge.net'> wrote: Am 03.08.2016 um 12:49 schrieb Ruga: > echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold > -w 999 | head -n 1 )" >example.txt > > spamassassin -t -D B_LLL.rule On Wed, Aug 3, 2016 at 12:15 PM, Axb <'axb.li...@gmail.com'> wrote >> please pastebin a sample msg signature.asc Description: OpenPGP digital signature
Re: Paragraph Length Limit (new rule)
OK Sent from ProtonMail Mobile On Wed, Aug 3, 2016 at 12:58 PM, Ryan Coleman <'ryan.cole...@cwis.biz'> wrote: Keep in mind we do not know that. It is better to not reply and wait a few hours than get Reindl worked up. :) > On Aug 3, 2016, at 5:55 AM, Ruga wrote: > > I am AWAY for my office. > Real spam truly unnecessary. > > Sent from ProtonMail Mobile > > > On Wed, Aug 3, 2016 at 12:51 PM, Reindl Harald <'h.rei...@thelounge.net'> > wrote: >> >> Am 03.08.2016 um 12:49 schrieb Ruga: >> > echo "$( cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold >> > -w 999 | head -n 1 )" >example.txt >> > >> > spamassassin -t -D B_LLL.rule > >> you where asked for a real *mail* example instead some generic stuff >> >> > On Wed, Aug 3, 2016 at 12:15 PM, Axb <'axb.li...@gmail.com'> wrote >> >> please pastebin a sample msg >>
Re: Paragraph Length Limit (new rule)
Use a 'full' not 'rawbody' rule. IE: full B_PLL /(?:(?!<\/p>).){999,}<\/p>/msi Why are you doing a "tflags __B_PLL multiple maxhits=1" ? If you have "maxhits=1" what's the point of "multiple" at all? On Wed, 3 Aug 2016, Ruga wrote: Hello, We received a new type of spam, twice, and we are not willing to give them a third chance. The body includes a long html paragraph (...) of headlines from the news. The following works at the command line: perl -p0e 's/((?:(?!<\/p>).){999,}<\/p>)/-->$1<--/msig' example.eml perl -n0e '/((?:(?!<\/p>).){999,}<\/p>)/msig and print "--->$1<---"' example.eml The following SA rule, however, does not work at all: rawbody __B_PLL /(?:(?!<\/p>).){999,}<\/p>/msi tflags __B_PLL multiple maxhits=1 meta B_PLL __B_PLL describe B_PLL Body: Paragraph Length Limit score B_PLL 1.0 I would be most grateful if you could spot the but in the above rule. -- Dave Funk University of Iowa College of Engineering 319/335-5751 FAX: 319/384-0549 1256 Seamans Center Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527 #include Better is not better, 'standard' is better. B{
Re: Paragraph Length Limit (new rule)
> Use a 'full' not 'rawbody' rule. I do not need to parse the header. >Why are you doing a "tflags __B_PLL multiple maxhits=1" ? >If you have "maxhits=1" what's the point of "multiple" at all? To limit the number of possible matches to a single one.
Re: Paragraph Length Limit (new rule)
Why are you doing a "tflags __B_PLL multiple maxhits=1" ? If you have "maxhits=1" what's the point of "multiple" at all? On 03.08.16 12:31, Ruga wrote: To limit the number of possible matches to a single one. you did not get it. what you describe is the standard behaviour. you need "tflags multiple" when you want more than one match. using "tflags multiple maxhits=1" is completely useless. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Despite the cost of living, have you noticed how popular it remains?
Re: Paragraph Length Limit (new rule)
On 03.08.16 12:42, Ruga wrote: This is the stripped test. The number of characters is reduced to 72 from the original 999: make your own choice. The attached e-mail triggers the rule *if* I remove at least one "example" string. Perl: 5.22.2 SA: 3.4.1 rawbody B_PLL m|(?:(?!).){72,}|msi describe B_PLL Paragraph Length Limit score B_PLL 1.0 again. post the real sample of the spam We can't help you if you only post the rule - we don't know what you expect it to match. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
Re: Paragraph Length Limit (new rule)
On 3 Aug 2016, at 6:07, Ruga wrote: Hello, We received a new type of spam, twice, and we are not willing to give them a third chance. The body includes a long html paragraph (...) of headlines from the news. The following works at the command line: perl -p0e 's/((?:(?!<\/p>).){999,}<\/p>)/-->$1<--/msig' example.eml perl -n0e '/((?:(?!<\/p>).){999,}<\/p>)/msig and print "--->$1<---"' example.eml The following SA rule, however, does not work at all: rawbody __B_PLL /(?:(?!<\/p>).){999,}<\/p>/msi Will not hit an unclosed tag. May take a very long time to check against some very long messages with pathological (but not uncommon or spam-only!) HTML, due to the open-ended {999,}. However, I happen to have a message that should match your rule. It matches in perl directly, with the matched string being a 163301-byte HTML mess which also happens to be one line when decoded from quoted-printable, but it does not match as a SA 'rawbody' pattern. It DOES match if the rule is switched from 'rawbody' to 'full'. It is not clear to me why that change results in a match. I also constructed a message where the '' was only 650 characters after the '' and reduced the minimum length from 999 to 300, and that also matched as a rawbody rule. It seems that there's something breaking in SA when a 'rawbody' match is too long. I suspect a logical problem in how SA "chunks" a message for body and rawbody tests, but I haven't tracked down the details... Looks like a SA bug to me. For performance, the ability to match unclosed paragraphs, and working around that bug,a better solution is: rawbody __B_PLL /(?:(?!<\/p>).){999}/msi That will match the first 999 characters of a long HTML paragraph, no matter how long it is and whether or not it is ever closed. It also gets around whatever SA bug is blocking the very long matches. tflags __B_PLL multiple maxhits=1 Pointless. Why set the "multiple" flag if you're going to set "maxhits=1"??? meta B_PLL __B_PLL describe B_PLL Body: Paragraph Length Limit score B_PLL 1.0 I assume this is a placeholder for future combination with other rules, since the 'meta' as it stands is pointless and simply having a paragraph longer than 999 characters isn't inherently or heuristically spammy.
Re: Paragraph Length Limit (new rule)
Sent from ProtonMail Mobile On Wed, Aug 3, 2016 at 6:59 PM, Matus UHLAR - fantomas <'uh...@fantomas.sk'> wrote: On 03.08.16 12:42, Ruga wrote: >This is the stripped test. The number of characters is reduced to 72 >from the original 999: make your own choice. The attached e-mail >triggers the rule *if* I remove at least one "example" string. > >Perl: 5.22.2 >SA: 3.4.1 > >rawbody B_PLL m|(?:(?!).){72,}|msi >describe B_PLL Paragraph Length Limit >score B_PLL 1.0 again. post the real sample of the spam We can't help you if you only post the rule - we don't know what you expect it to match. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
Re: Paragraph Length Limit (new rule)
I cannot post the original spam message. Protonmail checks the outgoing messages: if they are spammy, then the sender is banned. I managed to send you a stripped version of the original without loosing my account. Be happy with it, because it is as close as you will ever get to the ugliness of the original. Sent from ProtonMail Mobile On Wed, Aug 3, 2016 at 6:59 PM, Matus UHLAR - fantomas <'uh...@fantomas.sk'> wrote: On 03.08.16 12:42, Ruga wrote: >This is the stripped test. The number of characters is reduced to 72 >from the original 999: make your own choice. The attached e-mail >triggers the rule *if* I remove at least one "example" string. > >Perl: 5.22.2 >SA: 3.4.1 > >rawbody B_PLL m|(?:(?!).){72,}|msi >describe B_PLL Paragraph Length Limit >score B_PLL 1.0 again. post the real sample of the spam We can't help you if you only post the rule - we don't know what you expect it to match. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
RE: Paragraph Length Limit (new rule)
Copy paste the entire message, including headers to pastebin.com which is the preferred action, as the spam filters on this list may well filter it out if sent as an email anyway. ...Kevin -- Kevin Miller Network/email Administrator, CBJ MIS Dept. 155 South Seward Street Juneau, Alaska 99801 Phone: (907) 586-0242, Fax: (907) 586-4588 Registered Linux User No: 307357 From: Ruga [mailto:r...@protonmail.com] Sent: Wednesday, August 03, 2016 11:22 AM To: Matus UHLAR - fantomas; users@spamassassin.apache.org Subject: Re: Paragraph Length Limit (new rule) I cannot post the original spam message. Protonmail checks the outgoing messages: if they are spammy, then the sender is banned. I managed to send you a stripped version of the original without loosing my account. Be happy with it, because it is as close as you will ever get to the ugliness of the original. Sent from ProtonMail Mobile On Wed, Aug 3, 2016 at 6:59 PM, Matus UHLAR - fantomas <'uh...@fantomas.sk'> wrote: On 03.08.16 12:42, Ruga wrote: >This is the stripped test. The number of characters is reduced to 72 >from the original 999: make your own choice. The attached e-mail >triggers the rule *if* I remove at least one "example" string. > >Perl: 5.22.2 >SA: 3.4.1 > >rawbody B_PLL m|(?:(?!).){72,}|msi >describe B_PLL Paragraph Length Limit >score B_PLL 1.0 again. post the real sample of the spam We can't help you if you only post the rule - we don't know what you expect it to match. -- Matus UHLAR - fantomas, uh...@fantomas.sk<mailto:uh...@fantomas.sk> ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
Re: Paragraph Length Limit (new rule)
>Looks like a SA bug to me. At last... > rawbody __B_PLL /(?:(?!).){999}/msi Well done. >> tflags __B_PLL multiple maxhits=1 >Pointless. Why set the "multiple" flag if you're going to set "maxhits=1"?? To really stop at the first match. > simply having a paragraph longer than 999 characters isn't inherently or > heuristically spammy. We receive spam with long paragraphs of nonsensical garbage, possibly to poison the corpus and results of automated learning. They are now sending headlines from the news, instead of random words and broken grammar. By rejecting such junk upront, we keep the system healthy. Also, when you see such spam on screen, that single long paragraph that fills pages and pages with junk, it would be pointless to keep you as a postmaster...
Re: Paragraph Length Limit (new rule)
On Wed, 3 Aug 2016, Ruga wrote: Pointless. Why set the "multiple" flag if you're going to set "maxhits=1"?? To really stop at the first match. That is the behavior if the "multiple" flag is not provided. "multiple" exists specifically to allow you to *NOT* stop at the first match. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Individual liberties are always "loopholes" to absolute authority. --- 2 days until the 281st anniversary of John Peter Zenger's acquittal
Re: Paragraph Length Limit (new rule)
New tests lead me to the following rule. It works and is now deployed on production servers. full B_PLL /(?:(?!<\/p>).){2000}/msi describe B_PLL Paragraph Length Limit score B_PLL 1.5 rawbody has a hidden bug: it breaks the above rule too. I am re-writing all local rules to "full" until "rawbody" is fixed. 2000 is an arbitrary number that fits the local corpus at this time. The long paragraph in the original spam had 10797 characters, and was 192 lines long. The score is now 1.5/5.0. The original spam scored 1.6/5.0. An additional rule scores 1.0 for any uri to a php page, and a third rule scores 1.0 when the From addr contains numbers. The resulting score for the original spam is now 5.0/5.0 Please catch the bug in rawbody.
Re: Paragraph Length Limit (new rule)
So your script dings my websites because I use .php as an extension without doing SEO? That seems really silly. Many websites use internal pages without SEO because of the royal PITB they can be to program all the little variables. For crying out loud most unsubscribe links are scripts with variables! > On Aug 3, 2016, at 4:07 PM, Ruga wrote: > > An additional rule scores 1.0 for any uri to a php page,
Re: Paragraph Length Limit (new rule)
On Wed, 3 Aug 2016, Ryan Coleman wrote: So your script dings my websites because I use .php as an extension without doing SEO? That seems really silly. Many websites use internal pages without SEO because of the royal PITB they can be to program all the little variables. For crying out loud most unsubscribe links are scripts with variables! Ruga doesn't say whether or not that is in combination with the excessively-long paragraph hit. On Aug 3, 2016, at 4:07 PM, Ruga wrote: An additional rule scores 1.0 for any uri to a php page, -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- The real opiate of the masses isn't religion; it's the belief that somewhere there is a benefit that can be delivered without a corresponding cost. -- Tom of "Radio Free NJ" --- 2 days until the 281st anniversary of John Peter Zenger's acquittal
Re: Paragraph Length Limit (new rule)
> On Aug 3, 2016, at 4:37 PM, John Hardin wrote: > > On Wed, 3 Aug 2016, Ryan Coleman wrote: > >> So your script dings my websites because I use .php as an extension without >> doing SEO? >> >> That seems really silly. Many websites use internal pages without SEO >> because of the royal PITB they can be to program all the little variables. >> For crying out loud most unsubscribe links are scripts with variables! > > Ruga doesn't say whether or not that is in combination with the > excessively-long paragraph hit. > Fair enough… You are correct on that count. >>> On Aug 3, 2016, at 4:07 PM, Ruga wrote: >>> >>> An additional rule scores 1.0 for any uri to a php page,
Re: Paragraph Length Limit (new rule)
Am 03.08.2016 um 21:22 schrieb Ruga: I cannot post the original spam message. Protonmail checks the outgoing messages: if they are spammy, then the sender is banned that is why you where asked *as first response* to put it on pastebin signature.asc Description: OpenPGP digital signature
Re: Paragraph Length Limit (new rule)
On 3 Aug 2016, at 17:07, Ruga wrote: New tests lead me to the following rule. It works and is now deployed on production servers. full B_PLL /(?:(?!<\/p>).){2000}/msi describe B_PLL Paragraph Length Limit score B_PLL 1.5 rawbody has a hidden bug: it breaks the above rule too. That is consistent with my guess about where the bug arises. For 'rawbody' rules, messages are sliced into 1-2KB pseudo-paragraphs and apparently there's no mechanism to allow a rawbody rule to span an arbitrary number of those pieces. I am re-writing all local rules to "full" until "rawbody" is fixed. There's a strong possibility that this only arises when using look-aheads that span more than 2 pseudo-paragraphs, i.e. match >1024 characters. 2000 is an arbitrary number that fits the local corpus at this time. The long paragraph in the original spam had 10797 characters, and was 192 lines long. The score is now 1.5/5.0. The original spam scored 1.6/5.0. An additional rule scores 1.0 for any uri to a php page, and a third rule scores 1.0 when the From addr contains numbers. The resulting score for the original spam is now 5.0/5.0 Please catch the bug in rawbody. If you want to help make that more likely to happen, please create a bug report at https://bz.apache.org/SpamAssassin/