Re: CPAN: Ownership of the XML-XPathScript modules

[Matt Sergeant]

Yeah totally fine by me. Do I need to do anything in pause?

> On Jul 24, 2020, at 5:57 PM, Chris Prather  wrote:
> 
> 
> I talked to Matt via Facebook about handing over his XML modules 
> (Specifically XML::Parser) which is why I have first come I suspect now. I 
> wanted to make sure that they had people who were still interested in XML and 
> Perl shepherding them and I was happy to be that person.
> 
> I'm entirely ok with Yanick taking it if he's happy with that or with  
> Dominique taking it if they prefer. Otherwise I'm happy to shepherd it as 
> well. Basically whatever everyone else wants to do as long as they don't fall 
> into ADOPTME I'm good with :)
> 
> -Chris
> 
>> On Fri, Jul 24, 2020 at 3:53 PM Dominique Quatravaux 
>>  wrote:
>> I am DOMQ and I approve of this plan.
>> 
>> -- 
>> Dominique Quatravaux
>> domini...@quatravaux.org
>> 
>>> Le ven. 24 juil. 2020 à 21:34, Neil Bowers  a 
>>> écrit :
>>> Hi,
>>> 
>>> I’m one of the PAUSE admins, and I’m emailing you wearing that hat.
>>> 
>>> The XML-XPathScript distribution was originally created by Matt; Dominique 
>>> then did 5 releases; since 2005 Yanick has done 24 releases. Chris 
>>> (PERIGRIN) doesn’t seem to have done any releases, but he has the 
>>> first-come indexing permission on the lead module (XML::XPathScript), and 
>>> YANICK has first-come on the others. The rest of you have co-maint on the 
>>> (other) modules.
>>> 
>>> The fractured ownership means that no single person could grant co-maint to 
>>> someone else, or transfer ownership to a new maintainer. PAUSE tries hard 
>>> now to not this happen[1], so I’m tidying up the historical cases.
>>> 
>>> The default rule is that whoever has first-come on the lead module should 
>>> get it on the rest, but I suspect here that it would make more sense for 
>>> YANICK to get first-come on XML::XPathScript — is that ok?
>>> 
>>> Cheers,
>>> Neil
>>> 
>>> [1] http://neilb.org/2020/07/24/inconsistent-permissions.html

Giving danga::socket to pause id: NML

[Matt Sergeant]

I’m only a comaint on the module. Happy to wait a bit for Brad to get back to 
someone about it. But I suspect he’s moved on too.

Re: [nodejs] error writing binary data to TCP socket.

[Matt Sergeant]

This has been fixed since. I don't know when. I have Node 9.3 and it
produces a Buffer.

On Tue, Jan 2, 2018 at 4:18 PM, Stephen James  wrote:

> Using electron I am trying to write out some bytes on a TCP socket. I am
> using Buffer.from to convert to buffer before calling write, but am still
> getting the above error. I have simplified it down to just creating an
> empty ArrayBuffer and calling Buffer.from
>
>
> var abuff = new ArrayBuffer(8 + encodedBuffer.length);
> console.log(Buffer.from(abuff));
> socket.write(Buffer.from(abuff));
>
>
> TypeError: Invalid data, chunk must be a string or buffer, not object at
> Socket.write (net.js:667:11)
>
>
> The console.log shows the following:
>
> Uint8Array(51) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
> 0, 0, 0, 0, 0]
>
> and not Buffer?
>
>
> Why would Buffer.from not create a Buffer?
>
>
> Versions in electron:
>
> ares:"1.10.1-DEV"
>
> atom-shell:"1.7.9"
>
> chrome:"58.0.3029.110"
>
> electron:"1.7.9"
>
> http_parser:"2.7.0"
>
> modules:"54"
>
> node:"7.9.0"
>
> openssl:"1.0.2k"
>
> uv:"1.11.0"
>
> v8:"5.8.283.38"
>
> zlib:"1.2.11"
>
> --
> Job board: http://jobs.nodejs.org/
> New group rules: https://gist.github.com/othiym23/9886289#file-
> moderation-policy-md
> Old group rules: https://github.com/joyent/node/wiki/Mailing-List-
> Posting-Guidelines
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+unsubscr...@googlegroups.com.
> To post to this group, send email to nodejs@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/nodejs/33bc035f-8074-4cc8-8d14-b0603570db7a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Job board: http://jobs.nodejs.org/
New group rules: 
https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
To post to this group, send email to nodejs@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/nodejs/CAPJ5V2b89TkNexUjuFcSA70iyXOwM%2BUczwh5Uf%2By3CossHJCdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: qpsmtpd-async weird ParaDNS lookup problem + code fix

[Matt Sergeant]

To be fair I don't do perl coding any more, so I'm happy for someone 
to take over ParaDNS maintainence.


ParaDNS-XS is in my SVN server. Happy to share that with anyone who wants 
it. It's basically ParaDNS using adns backend rather than Net::DNS. It's a 
bit hacky and probably doesn't work 100%


Matt.

On Fri, 1 Aug 2014, David Favor wrote:


Matt Sergeant wrote:
Do you have ParaDNS::XS installed? If not, try installing it. It's 
generally likely to be better than plain ParaDNS (and will be used 
automatically if it's installed).


There is no ParaDNS::XS on cpan... At least nothing returned for...

   http://search.cpan.org/search?query=paradnsmode=all

Unsure what this has to do with setting a list of servers for ParaDNS
to query. Whether pure perl or XS, likely there still has to be a set
of servers to query.

The fix I'm using right now is a slight patch to Qpsmtpd/PollServer.pm
which simply gives ParaDNS a list of servers to check...

   my @nameservers = qw/127.0.0.1 8.8.8.8 8.8.4.4/;

   my $obj = ParaDNS-new(
   nameservers = \@nameservers,
   finished = sub { $self-continue_read(); 
$self-run_hooks(connect); },

   # NB: Setting remote_info to the same as remote_host
   callback = sub { $conn-remote_info($conn-remote_host($_[0])); },
   host = $ip,
   );

This works like a charm.

Reading through the ParaDNS code, it's unclear to me how qpsmtpd can work
at all without a list of servers for ParaDNS to query.

Re: qpsmtpd-async weird ParaDNS lookup problem

[Matt Sergeant]

Do you have ParaDNS::XS installed? If not, try installing it. It's 
generally likely to be better than plain ParaDNS (and will be used 
automatically if it's installed).


On Thu, 17 Jul 2014, David Favor wrote:


I've been running qpsmtpd-async for years on all sorts of servers.

Likely I have something in DNS setup slightly wrong on a new server
I'm setting up + what's wrong escapes me.

The symptom is qpsmtpd-async hanging forever in the HELO sequence.

Both forkserver + prefork work fine + async works so much better,
I'd like to resolve this problem.

Here's an example of the problem...

Listen child making a Qpsmtpd::PollServer for 7.
11869 in config(plugins)
11869 config(plugins) returning (resolve_sender_host 
dont_require_anglebrackets rcpt_simple t...@newswire.net ch...@newswire.net 
d...@newswire.net supp...@newswire.net account...@newswire.net
sa...@newswire.net debr...@newswire.net maildir /cluster/clients/ivan-budimir 
%d/users/%l/Maildir) from cache

DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)
11869 (connect) running plugin: resolve_sender_host
11869 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost 
domain=localhost

11869 Plugin resolve_sender_host, hook connect returned DECLINED,
11869 in config(smtpgreeting)
11869 config(smtpgreeting) returning (Ready!) from cache

Notice this line emitted from ParaDNS...

  DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

Bind looks good...

  biz-net2# netstat -pluten | grep named
  tcp0  0 127.0.0.1:530.0.0.0:*   LISTEN 
10560684715/named
  tcp0  0 127.0.0.1:953   0.0.0.0:*   LISTEN 
10560734715/named
  tcp6   0  0 ::1:53  :::*LISTEN 
10560704715/named
  tcp6   0  0 ::1:953 :::*LISTEN 
10560744715/named
  udp0  0 127.0.0.1:530.0.0.0:* 
10560674715/named
  udp6   0  0 ::1:53  :::* 
10560694715/named


And host seems to work too...

  biz-net2# host localhost
  localhost has address 127.0.0.1
  localhost has IPv6 address ::1

  biz-net2# host 127.0.0.1
  1.0.0.127.in-addr.arpa domain name pointer localhost.

If I turn on ParaDNS debugging (export PARADNS_DEBUG=100) I just see the 
lookup fail again...


  100/2 [17636] dns lookup: Trying to resolve A: 127.0.0.1
  100/2 [17636] dns lookup: NS Query: 127.0.0.1 (60806)
  DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

biz-net2# perl -MParaDNS -e 'print $ParaDNS::VERSION\n'
2.0

Anyone have any suggestions?

Re: [nodejs] Re: file system API race conditions

[Matt Sergeant]

There is no performance downside in the failure mode. But obviously in the 
success mode there's two function calls vs one. But realistically the overhead 
of file system operations on spinning disks will far outweigh the extra API 
calls.

 On Feb 26, 2014, at 7:40 PM, Alex Kocharin a...@kocharin.ru wrote:
 
  
 This sounds like a very good idea... I wonder, is there any downside to it? 
 Performance maybe?
  
  
 26.02.2014, 09:20, Andrew Kelley superjo...@gmail.com:
 Actually, there is a solution! fs.link to the new file. This will fail with 
 EEXIST if it exists. And then fs.unlink on the old file.
  
 Everything works, life is good
 
 On Wednesday, February 26, 2014 12:07:29 AM UTC-5, Andrew Kelley wrote:
 I guess it comes down to the rename C function: 
 http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html
  
 Doesn't support no-overwrite flags. So there's not really a fix to this.
 On Wednesday, February 26, 2014 12:01:18 AM UTC-5, Andrew Kelley wrote:
 Oops, I see that fs.createWriteStream has the flags as an option. But I 
 think the point still stands for fs.rename.
 
 On Tuesday, February 25, 2014 11:56:44 PM UTC-5, Andrew Kelley wrote:
 fs.open has an important flag you can set to avoid race conditions: wx
  
 This allows you to open a file, only if it does not exist. This is more 
 correct than doing fs.stat and checking the result, because things could 
 have changed by the time you get the callback.
  
 Where is this API for fs.createWriteStream and fs.rename? These guys need to 
 have that option too.
  
 
 -- 
 -- 
 Job Board: http://jobs.nodejs.org/
 Posting guidelines: 
 https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
 You received this message because you are subscribed to the Google
 Groups nodejs group.
 To post to this group, send email to nodejs@googlegroups.com
 To unsubscribe from this group, send email to
 nodejs+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/nodejs?hl=en?hl=en
  
 --- 
 You received this message because you are subscribed to the Google Groups 
 nodejs group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to nodejs+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.
 -- 
 -- 
 Job Board: http://jobs.nodejs.org/
 Posting guidelines: 
 https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
 You received this message because you are subscribed to the Google
 Groups nodejs group.
 To post to this group, send email to nodejs@googlegroups.com
 To unsubscribe from this group, send email to
 nodejs+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/nodejs?hl=en?hl=en
  
 --- 
 You received this message because you are subscribed to the Google Groups 
 nodejs group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to nodejs+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Swftools-common] Building PDF2SWF only

[Matt Sergeant]

After configure it looks like you should be able to drop into the src
directory and do make pdf2swf.

Matt.

On Mon, Aug 13, 2012 at 11:05 AM, Pablo Beltran pbeltr...@gmail.com wrote:

 Hi,

 SWFTools is a collection of programs. How can I build the *pdf2swf*program 
 only?

 Thanks,
 Pablo



 ---
 SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend
 an existing subscription, please kindly point your favourite web browser
 at:http://lists.nongnu.org/mailman/listinfo/swftools-common

---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: [Swftools-common] Building PDF2SWF only

[Matt Sergeant]

It's the base library for swftools

Try going into lib and typing make before going into src and typing
make pdf2swf

Matt.



On 2012-08-14, at 12:03 PM, Pablo Beltran pbeltr...@gmail.com wrote:

That looks the right way.

Everything gone well until message below:

make[1]: Entering directory
`/cygdrive/e/alchemy/tutorials/alexmac-alcextra-635c730/swftools-0.9.2/src'
make[1]: *** No rule to make target `../lib/libbase.a', needed by
`pdf2swf'.  Stop.
make[1]: Leaving directory
`/cygdrive/e/alchemy/tutorials/alexmac-alcextra-635c730/swftools-0.9.2/src'
Makefile:153: recipe for target `foo' failed
make: *** [pdf2swf] Error 2

What is libbase.a?



2012/8/14 Matt Sergeant m...@hubdoc.com

 After configure it looks like you should be able to drop into the src
 directory and do make pdf2swf.

 Matt.

 On Mon, Aug 13, 2012 at 11:05 AM, Pablo Beltran pbeltr...@gmail.comwrote:

 Hi,

 SWFTools is a collection of programs. How can I build the *pdf2swf*program 
 only?

 Thanks,
 Pablo



 ---
 SWFTools-common is a self-managed list. To subscribe/unsubscribe, or
 amend an existing subscription, please kindly point your favourite web
 browser at:http://lists.nongnu.org/mailman/listinfo/swftools-common



---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: [Swftools-common] PDF2SWF Error: Couldn't create temporary font file (swftools 0.9.2)

[Matt Sergeant]

Well let's not be naive here. Lots of people are using swftools on the
server. If they are having trouble with temporary files there's a
large potential for security issues. Basically always use known secure
systems for creating temporary files. Don't invent your own.

On 2012-06-16, at 5:47 PM, List_Subs list_s...@mavdns.net wrote:

 On Fri, 15 Jun 2012 12:38:53 +0200
 a...@tomo.at wrote:

 I have a php script to start several processes of pdf2swf working on
 the same pdf-file.

 The question, 'why, would you wish to have several process working on
 the same file?', sort of pops into my head.. ;o)

 If started immediately after each other, only one process survives,
 all others die with Error: Couldn't create temporary font file
 logged.

 Can't say I'm surprised.  They are all attempting to create/access the
 same temporary file, and thus getting in each others way.  If they had
 different temporary files to work on, then the problem would simply go
 away.

 It seems to me this is kind of a race condition with faster version
 0.9.2,  because if I add 1 second delay before starting subsequent
 processes no errors occur.

 Which makes sense, if you think about it.

 So, since you yourself created the issue, and have in one way at least,
 resolved it, it begs the further question, what exactly is the issue
 you have? ;o)

 Regards,


 Chris.

 ---
 SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
 existing subscription, please kindly point your favourite web browser 
 at:http://lists.nongnu.org/mailman/listinfo/swftools-common

---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: [Swftools-common] pdf2swf - different size swf on different platforms

[Matt Sergeant]

On Sat, Jun 9, 2012 at 11:28 AM, Pablo Rodríguez oi...@web.de wrote:

 Hi Matt,

 I'm afraid that the Linux binary might not use compression for the
 output PDF. I experienced that before with swfc (not sure with pdf2swf
 [I can't remember it]). Which Linux distribution are you using? Did you
 installed it from a repository or by yourself? Which swftools version
 are you using?


I installed from source and made sure that the ./configure output pretty
much matched on both OSs (with platform differences obviously). It's Ubuntu
(I think 10.4?). swftools is 0.9.2 on both platforms.

Did you see the dots difference I posted most recently? I'm pretty sure
it's related to the dots, not compression, as swfdump gives a significantly
different output, and a visual inspection shows that the two SWFs are
different (with respect to the dots).


 Since wine is able to use the Windows version of swftools, you have an
 easy way to check whether pdf2swf for Windows gives the same size results.


Sure, but I think the dots is the problem, and so I'd like to know how to
get the same results as the Mac produced, otherwise I'm going to have to
switch to an ImageMagick/convert version for this app and use images.

Matt.
---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: [Swftools-common] pdf2swf - different size swf on different platforms

[Matt Sergeant]

On Sat, Jun 9, 2012 at 2:10 PM, List_Subs list_s...@mavdns.net wrote:

  Sure, but I think the dots is the problem, and so I'd like to know
  how to get the same results as the Mac produced, otherwise I'm going
  to have to switch to an ImageMagick/convert version for this app and
  use images.

 Did you try some of the images manipulation swutches, available to
 pdf2swf and pds2swf -s?


Yes I tried pretty much everything. I got one option which reduced the SWF
from 500KB to around 350KB, but that's still way larger than the one the
Mac produced. (it was the -G flag which did that reduction IIRC). It gets
really small if I do -s bitmap, but that kinda defeats the purpose.


 Also, from which ( or with which ) application are you rendering the
 final pdf?   Maybe it's better to tweak the pdf with someting like
 pdftk first, or render via the Python gfx module?


The PDF is downloaded from a Bank's web site, so I can't really modify it
(aside from running it through pdftk as you say - I've yet to try that -
but it doesn't really answer the question of why the two OSes running the
same version of pdf2swf produce different results).
---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

[Swftools-common] pdf2swf - different size swf on different platforms

[Matt Sergeant]

I've been trying to debug why an 80KB PDF generates a 500KB SWF file.

If I run pdf2swf on a Linux box that's the size I get.

If I run it on a Mac I get a SWF file slightly smaller than the PDF.

I thought maybe this was a font issue so I copied all my fonts from the Mac
to the Linux box (after converting from .dfont files), and still got the
same results.

Any ideas why I get such a big difference? Shall I put the output from the
conversions with verbose on somewhere for people to look at?

Matt.
---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: [Swftools-common] pdf2swf - different size swf on different platforms

[Matt Sergeant]

On Fri, Jun 8, 2012 at 11:45 AM, List_Subs list_s...@mavdns.net wrote:

 On Fri, 8 Jun 2012 11:08:37 -0400
 Matt Sergeant m...@hubdoc.com wrote:

  I've been trying to debug why an 80KB PDF generates a 500KB SWF file.
 
  If I run pdf2swf on a Linux box that's the size I get.
 
  If I run it on a Mac I get a SWF file slightly smaller than the PDF.
 
  I thought maybe this was a font issue so I copied all my fonts from
  the Mac to the Linux box (after converting from .dfont files), and
  still got the same results.
 
  Any ideas why I get such a big difference? Shall I put the output
  from the conversions with verbose on somewhere for people to look at?

 Why not.  Maybe link to the 'offending' pdf as well?


Sadly I can't as it's a bank statement.

Here's the two files:

http://www.sergeant.org/linux.pdf2swf.out (WARNING: 13MB)
http://www.sergeant.org/mac.pdf2swf.out (only 513KB)

Command line used:

  pdf2swf -v -v -v -z -T 9 -t -s storeallcharacters Td.pdf

Matt.
---
SWFTools-common is a self-managed list. To subscribe/unsubscribe, or amend an 
existing subscription, please kindly point your favourite web browser 
at:http://lists.nongnu.org/mailman/listinfo/swftools-common

Re: Qpsmtpd vs Haraka performance

[Matt Sergeant]

Aleksandar Lazic wrote:

Hi Matt,

On Die 13.09.2011 11:10, Matt Sergeant wrote:

I just benchmarked Qpsmtpd vs Haraka and thought some here might find
the results interesting:

http://baudehlo.wordpress.com/2011/09/13/node-js-is-fast/

(not meant as inflammatory, obviously I still have love for Qpsmtpd)

Matt.


How about to check the regex performance with a plugin like:

check_badmailfrom_patterns
check_badrcptto_patterns
rcpt_regexp

I think the main benefit of perl is his regex engine.


Regular expressions are much faster in Javascript (well, in V8). They 
also suck (lots and lots of features missing). But if you don't need 
advanced features then they are going to be faster.


Matt.

Qpsmtpd vs Haraka performance

[Matt Sergeant]

I just benchmarked Qpsmtpd vs Haraka and thought some here might find 
the results interesting:


http://baudehlo.wordpress.com/2011/09/13/node-js-is-fast/

(not meant as inflammatory, obviously I still have love for Qpsmtpd)

Matt.


FYI: Qpsmtpd plugins were: logging/warn, rcpt_all_ok[1], queue/do_nothing[2]

[1]: sub hook_rcpt { return OK }
[2]: sub hook_queue { return (OK, Queued) }

Re: qpsmtp-async forwarder

[Matt Sergeant]

The forwarders (I supply 
both forwarder, after DATA, and full proxy) in Haraka is fully async. 
Maybe time to learn some _javascript_ :)


  
  
Alister WestAugust 25, 2011 9:10 PM
  
  Hi qpsmtpd,I came across an entry 
in the mail-archive link where Chris Lewismentions he has customised
 qpsmtp-async for forwarding to a list ofother servers.http://www.nntp.perl.org/group/perl.qpsmtpd/2009/07/msg8919.htmlChris
 My qpsmtp-async forwarder has a config list of IPs (and ports).I
 am about to write something that does something very similar (aload-balancer
 for smtp). As I'm not very familiar with qpsmtp I waswondering if 
Chris is still active here and if he would mind sharinghis altered 
forwarder so I could use it as base for my project.Any other 
suggestions, or modifications to qpsmtp-async,plugins/..smtp-forward,
 etc. welcome.Cheers for the good work!~~c|_| 
alisterwest.com - mmm coffee!

Re: [PATCH] tweak QP's header handling for messages with no body

[Matt Sergeant]

Yup. I should check if Haraka does the right thing with this too :)

On Tue, 16 Aug 2011, Jared Johnson wrote:


True, I was led astray by the comment that seemed to indicate it was all
about headers.  if the intention of the comment is correct, then it should
probably if ( $in_header and ... ), whether or not the new regex is added.
That block would also need to be moved to after the block that sets
$in_header = 0 if we've reached the end, to avoid catching the blank line
that separates headers and newlines.

At any rate, Chris has a point -- I was actually thinking the other day
that code like that original deferral would be better of sitting in a
plugin, so I could muck around with it in a more straightforward manner in
order to, say, log what we did to our own database, or change it to a
rejection, or whatever :)

Anyway, this is all kind of minutia compared to the original patch, which
I feel I should remind everyone has to do with accepting legitimate mail
from Lotus software without inadvertently stripping the headers :)

-Jared


That line of code doesn't look at the headers though, just at the final
dot at the end-of-data.




Jared Johnson mailto:jjohn...@efolder.net
August 16, 2011 3:00 PM


There's already a special case for something similar to this:

# Reject messages that have either bare LF or CR. rjkaes noticed a
# lot of spam that is malformed in the header.

($_ eq .\n or $_ eq .\r)
and $self-respond(421, See
http://smtpd.develooper.com/barelf.html;)
and return $self-disconnect;

you could just make it ( /\r\r\n$/ or $_ eq .\n or $_ eq .\r )
maybe?
and update the link to point to a URL that explains both?

-Jared





Matt Sergeant mailto:m...@sergeant.org
August 16, 2011 11:28 AM


Yup there's a lot of this going around right now. Just to be explicit
though, the header lines end in \r\r\n. Worth rejecting the bloody
lot, frankly :)



Chris Lewis mailto:cle...@nortel.com
August 15, 2011 4:21 PM




As a FYI, I've been seeing bot-emitted spam that appears to have extra
\r at the end of _all_ header lines, and the qpsmtpd parser seems to
be treating all of it as part of the _body_.  IOW: except for the
received line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers.

This implementation is backrev (0.80 I think), and as it's only spam
from one particular bot, we don't care about that particular wierdness
enough to investigate further.  But it's worth being aware of.



Jared Johnson mailto:jjohn...@efolder.net
August 15, 2011 3:39 PM


Hi,

We got a bug report from someone using IBM's Lotus suite (I think for
both
their MUA and MTA). Their users would often send messages where all the
content was in the subject and they didn't bother sending any message
content. I'm not sure if it's due to an apparently uncommon behavior for
their particular MUA or their MTA, but every one of these messages was
coming through with data in a form that looked like this:

Subject: howdy\r\n.\r\n

Rather than including the blank line that one might expect to follow
headers, since it's required in the event that a message body is
present:

Subject: howdy\r\n\r\n.\r\n

The customer reported these messages were having their subjects
stripped;
additional testing indicted all existing headers were being stripped. It
looks like this is because the loop that processes message data in
Qpsmtpd::SMTP::data_respond() and creates a Mail::Header object which is
later used to write out the header in delivery, only works if a blank
line
exists after the header, e.g. the second form above. The following is
all
I could find in RFC 5322 that elaborated on this blank line, which
obviously must exist if a message body is included:

A message consists of header fields (collectively called the header
section of the message) followed, optionally, by a body. The header
section is a sequence of lines of characters with special syntax as
defined in this specification. The body is simply a sequence of
characters that follows the header section and is separated from the
header section by an empty line (i.e., a line with nothing preceding the
CRLF).

I read this as implicitly allowing the exclusion of this blank line if
there is no message body: the specification for the blank line is only
mentioned in the description of the body, which is itself described as
optional. Considering we haven't run into this bug in years of usage, I
assume it's unconventional to exclude the blank line, but it looks like
it
is legitimate syntax.

At any rate this was effecting multiple legitimate end users so we put
together the attached patch, which pulls the header building into its
own
sub which is then called inside the loop if we

Re: [PATCH] tweak QP's header handling for messages with no body

[Matt Sergeant]

Yup there's a lot of this 
going around right now. Just to be explicit though, the header lines end
 in \r\r\n. Worth rejecting the bloody lot, frankly :)








  
Chris LewisAugust 15, 2011 4:21 PM
  
  
As a FYI, I've been seeing bot-emitted spam that appears to have 
extra 
\r at the end of _all_ header lines, and the qpsmtpd parser seems to be 
treating all of it as part of the _body_. IOW: except for the received 
line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers.

This implementation is backrev (0.80 I think), and as it's only spam
 
from one particular bot, we don't care about that particular wierdness 
enough to investigate further. But it's worth being aware of.


  
Jared JohnsonAugust 15, 2011 3:39 PM
  
  Hi,We got a bug report from 
someone using IBM's Lotus suite (I think for boththeir MUA and MTA).
  Their users would often send messages where all thecontent was in 
the subject and they didn't bother sending any messagecontent.  I'm 
not sure if it's due to an apparently uncommon behavior fortheir 
particular MUA or their MTA, but every one of these messages wascoming



 through with data in a form that looked like this:"Subject: 
howdy\r\n.\r\n"Rather than including the blank line that one 
might expect to followheaders, since it's required in the event that
 a message body is present:"Subject: howdy\r\n\r\n.\r\n"The



 customer reported these messages were having their subjects stripped;additional



 testing indicted all existing headers were being stripped.  Itlooks
 like this is because the loop that processes message data inQpsmtpd::SMTP::data_respond()



 and creates a Mail::Header object which islater used to write out 
the header in delivery, only works if a blank lineexists after the 
header, e.g. the second form above.  The following is allI could 
find in RFC 5322 that elaborated on this blank line, whichobviously 
must exist if a message body is included:"A message consists of 
header fields (collectively called "the headersection of the 
message") followed, optionally, by a body.  The headersection is a 
sequence of lines of characters with special syntax asdefined in 
this specification.  The body is simply a sequence ofcharacters that
 follows the header section and is separated from theheader section 
by an empty line (i.e., a line with nothing preceding theCRLF)."I
 read this as implicitly allowing the exclusion of this blank line ifthere



 is no message body:  the specification for the blank line is onlymentioned



 in the description of the body, which is itself described asoptional.



  Considering we haven't run into this bug in years of usage, Iassume



 it's unconventional to exclude the blank line, but it looks like itis



 legitimate syntax.At any rate this was effecting multiple 
legitimate end users so we puttogether the attached patch, which 
pulls the header building into its ownsub which is then called 
inside the loop if we reach the blank lineindicating the header 
section is complete; otherwise, it's called outsideof the loop if we
 have no more message data, indicating the header sectionis 
complete.  Sorry I'm not putting this on a github fork, I still don'thave



 my git stuff together, I may never get around to it but I thought youguys



 might find this useful.-Jared

Re: [PATCH] tweak QP's header handling for messages with no body

[Matt Sergeant]

That line of code doesn't 
look at the headers though, just at the final dot at the end-of-data.


  
  
Jared JohnsonAugust 16, 2011 3:00 PM
  
  There's already a special case for 
something similar to this:# Reject messages that have either
 bare LF or CR. rjkaes noticed a# lot of spam that is malformed 
in the header.($_ eq ".\n" or $_ eq ".\r")and 
$self-respond(421, "Seehttp://smtpd.develooper.com/barelf.html")
and return $self-disconnect;you could just make it (
 /\r\r\n$/ or $_ eq ".\n" or $_ eq ".\r" ) maybe? and update the 
link to point to a URL that explains both?-Jared
  
Matt SergeantAugust 16, 2011 11:28 AM
  
  




Yup there's a lot of this 
going around right now. Just to be explicit though, the header lines end
 in \r\r\n. Worth rejecting the bloody lot, frankly :)















  
Chris LewisAugust 15, 2011 4:21 PM
  
  
As a FYI, I've been seeing bot-emitted spam that appears to have 
extra 
\r at the end of _all_ header lines, and the qpsmtpd parser seems to be 
treating all of it as part of the _body_. IOW: except for the received 
line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers.

This implementation is backrev (0.80 I think), and as it's only spam
 
from one particular bot, we don't care about that particular wierdness 
enough to investigate further. But it's worth being aware of.


  
Jared JohnsonAugust 15, 2011 3:39 PM
  
  Hi,We got a bug report from 
someone using IBM's Lotus suite (I think for boththeir MUA and MTA).
  Their users would often send messages where all thecontent was in 
the subject and they didn't bother sending any messagecontent.  I'm 
not sure if it's due to an apparently uncommon behavior fortheir 
particular MUA or their MTA, but every one of these messages wascoming
 through with data in a form that looked like this:"Subject: 
howdy\r\n.\r\n"Rather than including the blank line that one 
might expect to followheaders, since it's required in the event that
 a message body is present:"Subject: howdy\r\n\r\n.\r\n"The
 customer reported these messages were having their subjects stripped;additional
 testing indicted all existing headers were being stripped.  Itlooks
 like this is because the loop that processes message data inQpsmtpd::SMTP::data_respond()
 and creates a Mail::Header object which islater used to write out 
the header in delivery, only works if a blank lineexists after the 
header, e.g. the second form above.  The following is allI could 
find in RFC 5322 that elaborated on this blank line, whichobviously 
must exist if a message body is included:"A message consists of 
header fields (collectively called "the headersection of the 
message") followed, optionally, by a body.  The headersection is a 
sequence of lines of characters with special syntax asdefined in 
this specification.  The body is simply a sequence ofcharacters that
 follows the header section and is separated from theheader section 
by an empty line (i.e., a line with nothing preceding theCRLF)."I
 read this as implicitly allowing the exclusion of this blank line ifthere
 is no message body:  the specification for the blank line is onlymentioned
 in the description of the body, which is itself described asoptional.
  Considering we haven't run into this bug in years of usage, Iassume
 it's unconventional to exclude the blank line, but it looks like itis
 legitimate syntax.At any rate this was effecting multiple 
legitimate end users so we puttogether the attached patch, which 
pulls the header building into its ownsub which is then called 
inside the loop if we reach the blank lineindicating the header 
section is complete; otherwise, it's called outsideof the loop if we
 have no more message data, indicating the header sectionis 
complete.  Sorry I'm not putting this on a github fork, I still don'thave
 my git stuff together, I may never get around to it but I thought youguys
 might find this useful.-Jared

Re: thoughts about a new module called check_spammer_connect

[Matt Sergeant]

On Wed, 27 Jul 2011, Jared Johnson wrote:


That sounds like a pretty sweet configuration!


[Note if you are running qpsmtpd-async, as we do, it's not really
possible to route DNS queries differently for DNSBLs versus other DNS
queries qpsmtpd does.  ParaDNS doesn't handle paralleled DNS queries to
different servers well.]


when I first mucked around with the uribl plugin (see other threads), I
switched the non-async plugin to use Net::DNS::Async, but as far as I
could tell, N::D::A wasn't truly async, so I left the async plugin using
ParaDNS.  As it turns out, one of my associates since determined a way to
use Net::DNS::Async in a truly async fashion.  We're still using the
prefork daemon, but when we inevitably switch to async I'll probably try
to switch the async to N::D::A.  We also have a consolidated feed so the
multi-feed thing isn't too much of a concern, but N::D::A just seems more
straightforward.  If anyone is interested in switching to N::D::A now,
feel free to ping me and I can get the details on how to use it in a way
that doesn't break the true async plugin... I'm also curious if anyone
knows of reasons why this switch would not be such a good idea ;)


I have an XS version of ParaDNS too, which may be useful to people. It's 
in the ParaDNS subversion repository. We use it at work and it seems 
stable now.


However I'm unlikely to maintain much on Qpsmtpd now that Haraka has taken 
off.


Matt.

Re: thoughts about a new module called check_spammer_connect

[Matt Sergeant]

On Wed, 27 Jul 2011, David Nicol wrote:


On Wed, Jul 27, 2011 at 2:05 PM, Matt Sergeant m...@sergeant.org wrote:

However I'm unlikely to maintain much on Qpsmtpd now that Haraka has taken
off.

Matt.


how about a plugin adapter, so Haraka can use Qpsmtpd plugins or v/v?
That probably implies
either a node.js -- perl integration layer, a V8 -- perl
integration layer, or use of
a more arm's-length plugin architecture (like sendmail's Milter)
around both parts.


Haraka ships with both smtp_forward and smtp_proxy plugins (with tradeoffs 
for each), so there's no reason you wouldn't just run both, if you needed 
qpsmtpd plugins too.


Matt.

Re: mutt

[Matt Sergeant]

Peter Corlett mailto:ab...@cabal.org.uk
July 25, 2011 5:55 AM


On Sun, Jul 24, 2011 at 11:00:13PM +0100, James Laver wrote:
[...]

It's quite shameful for RIM, given their devices are basically designed as
email terminals with a few other features added on as an afterthought.

The iOS mail client is best described as adequate. It's arguably better
than Outlook, which seems to be the standard MUA these days.


Not even close - that's just selection bias.

http://www.campaignmonitor.com/stats/email-clients/ (caveats apply, etc).



__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Fwd: Failed: PAUSE indexer report KMCGRAIL/Mail-SpamAssassin-3.3.2.tar.gz

[Matt Sergeant]

Done. I passed primary maintainership to you, so you can sort it out from 
there.


I also did the other ones I had:

Made KMCGRAIL primary maintainer of Mail::SpamAssassin::EncappedMIME.

Made KMCGRAIL primary maintainer of Mail::SpamAssassin::HTML.

Made KMCGRAIL primary maintainer of Mail::SpamAssassin::MailingList.

Made KMCGRAIL primary maintainer of Mail::SpamAssassin::SHA1.

Made KMCGRAIL primary maintainer of Mail::SpamAssassin::TextCat.



On Wed, 22 Jun 2011, Kevin A. McGrail wrote:


Poor intern...

Please give karma ASAP and supply an intern for the next round.

module: Mail::SpamAssassin::HTML
   version: undef
   in file: Mail-SpamAssassin-3.3.2/lib/Mail/SpamAssassin/HTML.pm
status: Not indexed because permission missing. Current registered
primary maintainer is MSERGEANT. Hint: you can always find
the legitimate maintainer(s) on PAUSE under View
Permissions.

module: Mail::SpamAssassin::MailingList
   version: undef
   in file: Mail-SpamAssassin-3.3.2/lib/Mail/SpamAssassin/MailingList.pm
status: Not indexed because permission missing. Current registered
primary maintainer is MSERGEANT. Hint: you can always find
the legitimate maintainer(s) on PAUSE under View
Permissions.


Regards,
KAM

Re: Someone needs to take jwz aside...

[Matt Sergeant]

Simon Wistow wrote:

On Wed, Jun 08, 2011 at 10:57:56AM -0400, Matt Sergeant said:
   

I'm actually liking it more than CPAN for publishing and installing stuff.
The only weak area is lack of search.cpan.org.
 


My problem with npm is that it either tries to install stuff in some
random directory in my home directory or suggests, with a straight face,
that I chown -R $USER /usr/local
   


I think that has changed now with 1.0. Every install is into the current 
directory now.


I think you'll find it is pretty close to bundler now. Though the 
documentation is still lacking.


One really nice thing I like is if you're working on a project, you put 
your dependencies into the package.json in your project directory, and 
then just type npm install and it pulls in those dependencies 
automatically and makes them available to that project (and only that 
project).


Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Someone needs to take jwz aside...

[Matt Sergeant]

Hakim Cassimally wrote:

While Javascript-the-language is lovely (as you say, better in some
respects, worse in others, than Perl), that's only one part of the
story.  I've not followed Javascript-the-platform that closely (i.e.
anything much beyond jQuery) - what's your experience been like,
working with Node and other libraries?


It's ok. The standard library shipping with node is lacking in areas 
(there's no flock(), there's no getopts parser, and a bunch of other 
things), and right now it feels like it is stagnating while they get in 
true Windows support, but I've been surprised by the breadth of things 
available on npm (the equivalent of cpan).



Perl-Javascript is a really interesting migration path I'd not
considered, and I'm not sure it's faster! would convince me on its
own -- we all know there are faster languages than Perl.  But... JS
does have a significant advantages over, say, Perl-Haskell, as
Javascript is so widespread and therefore has many(devs, projects,
jobs).


OK let me put it another way than just faster - it's the fastest 
option for dynamic languages (with the possible exception of Lua-JIT) 
that has a sane (ALGOL-style) syntax, while still having a large user base.


My thinking behind Haraka is that a lot of web sites need a mail server 
with custom functionality, and their web site coders know Javascript, so 
why not provide them with an option to do stuff in JS for email.


Also the other huge thing for me is I've been writing async code in Perl 
now for years, and you always come across fighting with some library 
because it's got a blocking API so either you can't use it, or you take 
a risk and block the event loop while it runs (the same is true for 
Twisted or the Ruby one). That just doesn't happen in Node - everything 
is async.


Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Someone needs to take jwz aside...

[Matt Sergeant]

David Cantrell wrote:

It's the lack of a CPAN-a-like for any other language that keeps me
coming back to perl.

Of course, it's possible that the Comprehensive Python Archive Network
or similar for ruby/javascript/java/C/whatever does exist but I just
can't find it.  But then, if I can't find it, it's not much use.

http://npmjs.org/

I'm actually liking it more than CPAN for publishing and installing 
stuff. The only weak area is lack of search.cpan.org.


Note that this is for Node.js stuff only - not Javascript in general.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: [Fwd: Re: [Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]]

[Matt Sergeant]

Jared Johnson wrote:

I ... disagree.  From my reading of plugins/tls, it looks like there is
no problem at all, in the non-async code path.  It resets STDIN and
STDOUT to a socket created from scratch by the IO::Socket::SSL module.

I haven't looked at IO::Socket::SSL to see if it has this sort of
issue, but it seems unlikely to me.


Ah good news then. Hopefully someone will apply my pull request for that 
patch.


Matt.

Re: Speed v Version

[Matt Sergeant]

Dave Hodgkinson wrote:

My BBC sandbox is sane at least:

$ uname -p
x86_64
   


Shouldn't a BBC report 6502? ;-)

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Someone needs to take jwz aside...

[Matt Sergeant]

Simon Cozens wrote:

On 02/06/2011 21:50, gvim wrote:
   

  Considering the amount of development you've done on Perl web frameworks over
  the years isn't this tantamount to having given up on Perl, at least for web
  development?
 


Yes and no. I've moved from being more of a developer to being more of a user.
Perl is a fantastic language for developers. It has a great culture for
developers. We're all brilliant at producing tools which other developers can
pick up and do really great stuff with. Perl is wonderful if I want to write
my own web framework, or construct my own CMS on top of one of the hundreds of
Perl web frameworks which already exist.
   


As someone else who has written a bunch of popular perl stuff over the 
years, I'll chime in here too - I write a lot less open source stuff 
these days, but when I do I'm looking much more to JavaScript. The 
language is actually about as good as Perl (some areas better, some 
worse), but the implementation, the interpreters, are just WAY faster.


https://github.com/baudehlo/Haraka

Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: messagelabs.com contact - SMTP-side domaincheck checks IPv4 only, rejects domains with first MX on IPv6

[Matt Sergeant]

I'll get someone to contact Ford and see what they are running. From 
google it looks like Exchange. Is this a known bug with Exchange? If so 
I think there's bigger problems than messagelabs :)


Jeroen Massar wrote:

As the subject states,

mailer-dae...@messagelabs.com:

@ford.com:
Connected to 136.1.7.8 but sender was rejected.
Remote host said: 501 Sender domain must exist

As it obviously checks only the first MX record if there are A records,
and if there are none it rejects it. This while there are  records
on the first MX, and even A records on the remaining MXs. Thus a proper
fix would already be to check the other MXs and of course to check for
 too ;)

And that affects all customers at messagelabs, thus if somebody can pass
that along to them to fix it, that would be great ;)

Oh and of course the check is also there for postmaster@ thus no way to
tell them through that route.

Greets,
  Jeroen


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
__

   


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: [Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]

[Matt Sergeant]

No takers? I do consider the bug fairly minor (it's not like a remote 
root or anything)... But still?


Matt Sergeant wrote:

I'm forwarding this to the list since I didn't get a response from Ask...

The problem here is when someone sends the following packet:

STARTTLS\nSOME_COMMAND\n

The SOME_COMMAND bit gets cached internally (in PollServer/async 
that's in $qp-{line}, but in the other implementations I have no idea 
what happens) and then because SSL upgrading happens in external 
libraries, they don't see these bits of data, so what then happens is 
ssl gets upgraded, but then SOME_COMMAND gets executed after SSL has 
been negotiated.


I'm pretty sure this is enough to fix it for async:

diff --git a/plugins/tls b/plugins/tls
index 37fbc9a..f850d2c 100644
--- a/plugins/tls
+++ b/plugins/tls
@@ -275,6 +275,7 @@ sub upgrade_socket {
 my UpgradeClientSSL $self = shift;

 unless ( $self-{_ssl_started} ) {
+$self-{_stashed_qp}-clear_data();
 IO::Socket::SSL-start_SSL(
 $self-{_stashed_qp}-{sock}, {
 SSL_use_cert = 1,

I've submitted a pull request for that.

But for the non-async scenario I think it's a lot more complex because 
the caching would be done at the C-level, so a fix more like the fix 
(posted below) for postfix is required (switch to non-blocking, get 
whatever data is remaining, flush it, switch back off non-blocking).


 Original Message 
From: Graham Todd gt...@iciti.ca
Subject: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl 
qpsmtpd

Date: Wed, 01 Jun 2011 15:03:51 -0400
To: m...@fehcom.de, ga...@freebsd.org, Matt Sergeant 
m...@sergeant.org




Hi I'm a user of the qpsmtpd and qmail-spamcontrol ports on FreeBSD
(ga...@freebsd.org is the qmail-spamcontrol port maintainer), thanks to
you all for making qmail even more excellent by integrating all the
patches and making it easy to build and install on FreeBSD and using perl
to make it as flexible as Apache ;-)

As I understand it STARTTLS support in qmail-spamcontrol is handled by
ucspi-ssl with TLS patches. We're currently using qpsmtpd which has a
plugin that wraps qmail with perl based support for STARTTLS (using
IO::Socket::SSL).

Does Wietse Venema's STARTTLS vulnerability impact the STARTTLS
enhancements used in ucspi-ssl, qpsmtpd or qmail-spamcontrol? The
vulnerability is described here:

http://www.securityfocus.com/bid/46767

There is a fairly simple patch that fixes the TLS vulnerability and
applies to netqmail-1.06-tls - I am wondering how the vulnerabilty might
effect TLS functions in qpsmtpd and qmail-spamcontrol. The patch for TLS
enhanced versions of vanilla qmail-1.03 is here:

http://marc.info/?l=qmail-ldapm=130564281418177w=2

There's a long list of spots to look for the vulnerability on the 
Security

Focus site but neither qpsmtpd nor ucspi-ssl-tls are mentioned.

Cheers,

gtodd 

[Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]

[Matt Sergeant]

I'm forwarding this to the list since I didn't get a response from Ask...

The problem here is when someone sends the following packet:

STARTTLS\nSOME_COMMAND\n

The SOME_COMMAND bit gets cached internally (in PollServer/async that's 
in $qp-{line}, but in the other implementations I have no idea what 
happens) and then because SSL upgrading happens in external libraries, 
they don't see these bits of data, so what then happens is ssl gets 
upgraded, but then SOME_COMMAND gets executed after SSL has been negotiated.


I'm pretty sure this is enough to fix it for async:

diff --git a/plugins/tls b/plugins/tls
index 37fbc9a..f850d2c 100644
--- a/plugins/tls
+++ b/plugins/tls
@@ -275,6 +275,7 @@ sub upgrade_socket {
 my UpgradeClientSSL $self = shift;

 unless ( $self-{_ssl_started} ) {
+$self-{_stashed_qp}-clear_data();
 IO::Socket::SSL-start_SSL(
 $self-{_stashed_qp}-{sock}, {
 SSL_use_cert = 1,

I've submitted a pull request for that.

But for the non-async scenario I think it's a lot more complex because 
the caching would be done at the C-level, so a fix more like the fix 
(posted below) for postfix is required (switch to non-blocking, get 
whatever data is remaining, flush it, switch back off non-blocking).


 Original Message 
From:   Graham Todd gt...@iciti.ca
Subject:STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd
Date:   Wed, 01 Jun 2011 15:03:51 -0400
To: m...@fehcom.de, ga...@freebsd.org, Matt Sergeant m...@sergeant.org



Hi I'm a user of the qpsmtpd and qmail-spamcontrol ports on FreeBSD
(ga...@freebsd.org is the qmail-spamcontrol port maintainer), thanks to
you all for making qmail even more excellent by integrating all the
patches and making it easy to build and install on FreeBSD and using perl
to make it as flexible as Apache ;-)

As I understand it STARTTLS support in qmail-spamcontrol is handled by
ucspi-ssl with TLS patches. We're currently using qpsmtpd which has a
plugin that wraps qmail with perl based support for STARTTLS (using
IO::Socket::SSL).

Does Wietse Venema's STARTTLS vulnerability impact the STARTTLS
enhancements used in ucspi-ssl, qpsmtpd or qmail-spamcontrol? The
vulnerability is described here:

http://www.securityfocus.com/bid/46767

There is a fairly simple patch that fixes the TLS vulnerability and
applies to netqmail-1.06-tls - I am wondering how the vulnerabilty might
effect TLS functions in qpsmtpd and qmail-spamcontrol. The patch for TLS
enhanced versions of vanilla qmail-1.03 is here:

http://marc.info/?l=qmail-ldapm=130564281418177w=2

There's a long list of spots to look for the vulnerability on the Security
Focus site but neither qpsmtpd nor ucspi-ssl-tls are mentioned.

Cheers,

gtodd

Re: smtp proxy to external smtp server

[Matt Sergeant]

What do you mean by signed?

Do you mean like adding a banner to the text parts of the email? If so, 
that's a really hard problem (I mean it's doable in simple situations, 
but breaks very very easily).


Mike Korizek wrote:

On 05/17/2011 04:24 PM, Matt Sergeant wrote:
   

It can be done, but you'll need to customise the smtp-forward plugin
yourself to do it.
 

I checked the transaction object, I could not find a handle to the message.
How can I achieve the following:
An email shall be signed and then put back to the queue.
Before signing the email I parse it with a MIME::Parser, but how can I
put the new email back to the queue?
Thanks for any hint.
Mike

   

Re: smtp proxy to external smtp server

[Matt Sergeant]

Aleksandar Lazic wrote:

Just for my curiosity, why don't you use

qpsmtpd::smtp-forward =Any MTA Setup (postfix,courier,qmail, ...)? 


It's not sender dependent, and doesn't pass on AUTH. (but would be 
easily hackable to do that).

Re: Xcode 4.0.2 and XS modules

[Matt Sergeant]

Nicholas Clark wrote:

Would they care more if they got lots of polite bug reports from registered
developers who care about Apple's SNAFU, encouraging Apple to re-instate the
PPC assembler for XCode 4? Is XCode 4*supposed*  to support the PPC-enabled
OS X versions? Or is it Lion only?
   

It's supposed to remove PPC support in preparation for Lion.


Also, did this bug start with XCode 4.0.whatever-the-first-one-is, or was
the bug added/the functionality removed in a point release?


I downloaded a beta version (I'm a paid-up ADC member) and it was there.

I've filed a report with apple. Bug Problem ID: 9340360

I noted on the bug report that you can fix it. Just edit -arch ppc out 
of Config_heavy.pl


Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Someone needs to take jwz aside...

[Matt Sergeant]

Abigail wrote:

On Wed, Apr 20, 2011 at 06:43:48PM +0200, Lars Thegler wrote:
   

On Wed, Apr 20, 2011 at 10:40 AM, Dave Hodgkinsondaveh...@gmail.com  wrote:
 

http://www.jwz.org/blog/2011/04/a-badge-for-the-software-industrys-failures/

Or does he have a point?
   

No, code reuse is a *good* thing.
 


Sometimes.

But I've seen so much code that's needlessly convulated just so it can
just some code (or worse, that the code can be reused), that I take the
dogma code reuse with more salt than the yearly recommended dosis.
   
Especially when the code reuse in this case is JUST to get access to 
strptime.


Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Announce: Haraka

[Matt Sergeant]

Some of you may be interested in this...

I decided I wanted to hack on node.js to see what all the fuss is about.

So to do that I have basically ported Qpsmtpd to Node.js (and given it a 
decent name while doing so!).


It's still early days - there are no plugins to speak of yet (i.e. no 
queue plugins at all yet), but you might be interested in just looking 
anyway:


https://github.com/baudehlo/Haraka

Re: Announce: Haraka

[Matt Sergeant]

Guy Hulbert wrote:

  https://github.com/baudehlo/Haraka
 


Had a look.  I recognize bits.  Do you have any feeling for how easy it
is to code versus perl


Once you get used to the idiosyncrasies of Javascript, just as easy 
really. Took me a while to understand the object model, but everything 
seems to work ok now I mostly figured it out.


There's still stuff I have no idea how to do (the lack of caller 
telling me line numbers is annoying for example), but I'm getting there.


The other big thing is the libraries are simultaneously both sparse and 
weak. There's a CPAN equivalent (called npm) but it's still very 
immature in terms of what's available. Even the core libraries are 
missing some fundamental stuff (like I can't do TLS as they removed 
support for upgrading a single client connection to TLS).



  and how the performance might be versus qpsmtpd.
   


It pretty much blows it away. Node can be pretty much as fast as a lisp 
or smalltalk implementation. All that research into dynamic languages is 
all going into Javascript these days, and none is going into perl (some 
is going into Python and Ruby, but not as much as JS). So I think it 
will always be quite a bit faster.


For reference I can (albeit without plugins) throw about 5000 mails/sec 
through it fairly easily it seems, though that's with persistent 
connections, and on localhost.


Performance was my main motivation for doing it.

The other thing is that everyone doing node.js is doing event 
programming, so all those libraries for database access and 
memcached/redis access and so on are all async already.


Matt.

Re: check_badrcptto, to prevent backscatter

[Matt Sergeant]

Should we have plugins/qmail and plugins/postfix dirs?

Todd Brunhoff wrote:
 Tim's view seems appropriate. His script is centered on qmail, and 
mine is centered on postfix (or more specifically, on /etc/aliases). 
Both scripts are probably best in their current form with appropriate 
disclosures. Let me know if there is any prep work you would like me 
to do to my script.


Todd

On 3/9/2011 7:23 AM, Tim Meadowcroft wrote:

On Sunday 06 March 2011 06:42:50 Robert Spier wrote:

Todd Brunhoff wrote:

   Your scripts look like they have a good deal of qmail
   sophistication. Some years ago I ran qmail 1.0.3, after each major
   system crash, I would revisit whether to use qmail, and eventually
   decided to switch to qpsmtp+postfix because both seem to have 
better

   support.  And in fact, the reason I included /etc/alias was to
   replace the very useful alias mechanism in qmail. I really didn't
   need much, so that was sufficient for me.

So it seems that among these collections of scripts there are
backscatter solutions for qmail sites and qpsmtp sites. Perhaps one of
the developers can fold these into a contrib folder? 

A lot of plugins are linked from the wiki, http://wiki.qpsmtpd.org.

check_goodrcptto looks like something that might be worth having in
core.  Tim and Todd, are you interested in reconciling the differences
between your versions?  (Maybe some sort of configuration interface?) 
Mine is very qmail specific - I sort of feel generalising it would 
make it
worse. Either you use qmail, in which case you might like it as it 
stands,
(possibly in addition to other recipient checks) or you don't use 
qmail, in

which case you can ignore it completely.

I've posted the source (with disclaimers - I'm still on qpsmtpd 
v0.28) at


   http://schmerg.com/checkgoodrcptto-a-qpsmtpd-plugin-for-checking

and will see about adding it to the wiki when I can set up an account 
there.


Cheers

--
Tim 

Re: rolling uribl and rbl plugins into a single plugin

[Matt Sergeant]

Jared Johnson wrote:

So our organization is planning doing some big changes to the rbl plugin
and it dawned on us that it seems a lot easier to just add an earlier hook
to the existing uribl plugin (and rename it to rbl?  or bl? or
something?).  But of course I still have in mind that someday I'll get the
plugin completely in shape and it will be in QP, and we'll get to share
code and all.  Would this be an undesirable direction to take the uribl
plugin in for QP proper, in the event that the uribl plugin was integrated
into QP proper?  It seems like it would be handy in terms of both code
organization and features (the main feature I can think of is proper mask
evaluation along with custom actions per mask, just like in the uribl
plugin).

Thoughts?
   
I'd much prefer they both used plugin inheritance to access shared code 
than have them merged.

Re: rctpto rejection based on smtp verify?

[Matt Sergeant]

Charlie Brady wrote:

On Fri, 14 Jan 2011, Nicholas Lee wrote:
   

Is there a plugin that will check rcpt addresses against a back end smtp
server?
 


I presume you intend to eventually deliver to the same smtp backend, via
the smtp-forward plugin.

I've long argued that the smtp-forward plugin should hook into different
phases of the transaction, and do the recipient validation directly
against the backend:

http://www.nntp.perl.org/group/perl.qpsmtpd/2008/11/msg8288.html
   


I think it should be optional, but yeah. There are downsides (such as 
keeping open the SMTP connection with the backend), but it'd be nice if 
you could choose.


Matt.

Re: Recommendation for simple Web Frameworks

[Matt Sergeant]

Eden Cardim wrote:

Simon == Simon Wistowsi...@thegestalt.org  writes:


 Simon  In short - I don't really need the CRUD stuff from a
 Simon  framework, I really just need the url based dispatch. I
 Simon  played around with Catalyst (which I'm familiar with from 6A)
 Simon  but it felt like it was a bit of a sledgehammer and that it
 Simon  was (not unreasonably) tied to an ORM.

You got the wrong impression, Catalyst isn't tied to anything, it's just
a plugglable/configurable http dispatcher, it's as simple as any of the
other perl web frameworks in that regard. It does have a larger user
base for testing/contribution/support, I'd say that's the main
advantage.


But the dependencies list *is* much larger for Catalyst.

http://deps.cpantesters.org/?module=Dancer;perl=latest

vs

http://deps.cpantesters.org/?module=Catalyst::Runtime;perl=latest

Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: sudden low spam levels?

[Matt Sergeant]

Ken Chase wrote:

I have two independent mailservers, and two other customers that run their own
servers, all largely unrelated infrastructures and target domains, suddenly
experiencing low levels of spam.

Total emails/day dropping from some 175,000-250,000ish to 50-75,000ish (legit
mail in the 2-5,000 per day, yes I have some high spam:legit customers...). 3
days in a row now at least, at quick glance.

Did someone set up them the bomb?
   


Something killed off RuStock at Xmas.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: AnyEvent mode?

[Matt Sergeant]

Aleksandar Lazic wrote:

On Mon 06.12.2010 16:34, Matt Sergeant wrote:

Aleksandar Lazic wrote:

Do you have benchmarked it with smtpstone from postfix or some other
tools? 


I just threw it on our spamtrap which does approx 50m emails/day. 


Do you really mean 50 Million?


Yes.

Wow that's a lot ;-) 


Not compared to another qpsmtpd spam trap I know of it's not. :-)

Matt.

Re: AnyEvent mode?

[Matt Sergeant]

Aleksandar Lazic wrote:

Do you have benchmarked it with smtpstone from postfix or some other
tools? 


I just threw it on our spamtrap which does approx 50m emails/day.

Re: AnyEvent mode?

[Matt Sergeant]

On Thu, 2 Dec 2010, Aleksandar Lazic wrote:


On Don 02.12.2010 19:04, Matt Sergeant wrote:

On Thu, 2 Dec 2010, Ask Bjørn Hansen wrote:



On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote:


Maybe we can make another benchmark AnyEvent vs  Danga::Socket due to
the fact that AnyEvent with EV as underlaying event lib looks very fast
from the internet source ;-)


Matt was (I'm guessing) testing a load that's artificial to anyone who's 
not just archiving spam from a spam trap.  For the rest of us by far most 
of the resources go to the various spam filtering stuff, so the 
performance will be fine.


More important is the ease of use of the APIs, the general eco system etc. 
On those AnyEvent wins (in my opinion).


Sorry yeah I think I have it working, but our work data centre is down
right now so I can't get at the files :-(

Will email the latest ones here when I get the chance.


Thanks, I'am quite interested.


OK, the relevant files are attached. Nothing else needed changing I don't 
think.


AnyEvent.pm has to go in the lib/Qpsmtpd/ dir.#!/usr/bin/perl

use lib ./lib;
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}

# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;

use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib $FindBin::Bin/lib;
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;

$|++;

$SIG{'PIPE'} = IGNORE;  # handled manually

$DEBUG  = 0;

my $PORT= 2525;
my $LOCALADDR   = '0.0.0.0';
my $PROCS   = 1;
my $USER= (getpwuid $)[0]; # user to suid to
   $USER= smtpd if $USER eq root;
my $PAUSED  = 0;
my $NUMACCEPT   = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH;   # daemonize on startup

# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX = 1000;

sub reset_num_accept {
$NUMACCEPT = 20;
}

sub help {
print EOT;
Usage:
qpsmtpd [OPTIONS]

Options:
 -l, --listen-address addr : listen on a specific address; default 0.0.0.0
 -p, --port P  : listen on a specific port; default 2525
 -u, --user U  : run as a particular user; defualt 'smtpd'
 -j, --procs J : spawn J processes; default 1
 -d, --detach  : detach from controlling terminal (daemonize)
 --pid-file P  : print main servers PID to file P
 
 -h, --help: this page
EOT
exit(0);
}

GetOptions(
'p|port=i'  = \$PORT,
'l|listen-address=s'= \$LOCALADDR,
'j|procs=i' = \$PROCS,
'v|verbose+'  = \$DEBUG,
'u|user=s'  = \$USER,
'pid-file=s'= \$PID_FILE,
'd|detach'  = \$DETACH,
'h|help'= \help,
) || help();

# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { help }

use constant READY  = 1;
use constant ACCEPTING  = 2;
use constant RESTARTING = 999;

my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;

if ($PID_FILE  -r $PID_FILE) {
open PID, $PID_FILE
or die open_pidfile $PID_FILE: $!\n;
my $running_pid = PID || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die Found an already running qpsmtpd with pid $running_pid.\n;
}
}
close(PID);
}

run_as_server();

exit(0);

sub spawn_child {
my ($plugin_loader) = @_;

my $pid = fork;

if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die Unable to fork;
}

$plugin_loader-run_hooks('post-fork');

accept_loop();
}

sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print Connect?\n;
while ($SERVER  (my $peer = accept my $fh, $SERVER)) {
 fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside 
world does not

 my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
 my $qp = Qpsmtpd::AnyEvent-new($fh, format_address($host), 
$service);
 $qp-process_line(Connect);
  }
};

AnyEvent-condvar-wait;
exit;
}

sub sig_hup {
kill 1, keys %childstatus;
}

sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG))  0) {
if (!defined $childstatus{$child}) {
next;
}

last unless $child  0;
print SIGCHLD: child $child died\n;
delete $childstatus{$child};
$CURRENT_PROCS--;
}

$SIG

Re: Exim alternatives?

[Matt Sergeant]

I once wrote a very simple perl module which basically did outbound mail 
queueing for very simple needs... But then I discovered my needs were a 
bit more complex. LOL...


I second Ask's question - what's wrong with using exim?

David Favor wrote:

I'm looking for a simple alternative to exim, sendmail, postfix
for outgoing email.

So when and email comes into qpsmtpd which forwards offsite,
an outgoing mail server to forward these email.

If someone has a suggestion about other options, please let
me know.

Thanks. 

Re: Exim alternatives?

[Matt Sergeant]

On Tue, 14 Sep 2010, David Favor wrote:


The problem is two fold.

1) I could never get a straight answer about the correct
  configuration from the exim folks.

2) The config I have works for weeks to months, then develops
  odd (bitrot) challenges which seem to relate to DNS MX server
  changes in domains, which exim never senses.

  For example, message is queued and can't reach an MX record,
  then continually tries the bad MX record, never attempting
  to look up a new record.

I'm switching back to TipJar::MTA to do some testing.

What I require is a simple system, preferably perl based, so it's
fairly easy to understand + configure... that simply works... all
the time... no fuss, no muss.


OK... try my MrQueue stuff:

svn co svn://axkit.org/MrQueue

(axkit.org is having a bit of trouble resolving right now, so try 
sergeant.org if that doesn't work - same host).


Matt.

Re: [Bug 6483] request to use RE2 in place of RE2C step

[Matt Sergeant]

bugzilla-dae...@issues.apache.org wrote:

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6483

--- Comment #2 from Mark Martinecmark.marti...@ijs.si  2010-08-18 10:46:44 
UTC ---
   

Is there a mature perl wrapper for RE2?
 


http://github.com/dgl/re-engine-RE2

(it's pretty fresh, don't know how mature it is)
   


It works, and allows it to be a drop-in replacement for perl's regexp 
engine, so all you'd need to do is check if it can be loaded, and the 
perl version is high enough (5.10 required), and support it in the rule 
compiler (not even the re2c stuff).


There's some minor bugs in the UTF-8 support apparently. I've asked 
David if he wants to comment here.


Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Rewritten URIBL plugin

[Matt Sergeant]

Jared Johnson wrote:

sub parse_mime {


That works, only this should be called parsed_mime because you're 
asking for the parsed bit, not telling it to parse (every time).


Matt.

Re: Rewritten URIBL plugin

[Matt Sergeant]

On Sun, 25 Jul 2010, Jared Johnson wrote:


The plugin has the following advantages over the original:

- Uses MIME::Parser to unpack message text so that we can look for URI's
in base64-encoded data, etc., and _not_ look for URI's in noise.


I think we should probably consider putting support for parsed messages 
into core, with the parsing done lazily if requested by the API.


Thoughts?

Matt.

Re: SPF +all

[Matt Sergeant]

FWIW we've told Nationwide about this. They are including 
include:messagelabs.com which should be spf.messagelabs.com. PEBCAK.


John Hardin wrote:
There's a thread that's currently on the users list about Nationwide 
Bank in UK publishing an SPF record that includes messagelabs, and 
messagelabs' SPF record says +all. This makes it a little difficult 
to use SPF to reject phishing.


In light of that, do we want to revisit 
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5684 and 
reconsider it for 3.3.x or 3.4.x? 


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Sanity check - DateTime on Perl 5.12.1

[Matt Sergeant]

Toby Wintermute wrote:

Hi,
I wondered if anyone else running Perl 5.12.1 (and 12.0 would be
interesting too) could quickly check if they can build DateTime 0.5x
and pass the unit tests?

I have them failing on two machines, but they're very similar and I am
worried I might have screwed something up elsewhere, since I have been
playing with a lot of versions of Perl on said machines.
   


You know that Perl 5.12 ships with a pretty awesome date/time library, 
right? ;-)


(yeah I know it doesn't do everything DateTime does, it's a good 
lightweight alternative)


Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Redirecting email

[Matt Sergeant]

Chris Lewis wrote:


The version I've derived from Steve's works during hook_rcpt.


Yes, but assigning to $_[0] is a horrible way to do it. We should have a 
supported and cleaner way to modify the email address at RCPT TO time...


Ah, we do have a supported way... a hook called rcpt_pre which returns 
OK, $new_address (with angle brackets).


Given the way my qpsmtpd works, hook_data is way way too late.  Too 
much has already been decided by that point. 


I don't mean data_post - I mean right on the DATA command, which happens 
immediately after the RCPT TOs in a normal SMTP transaction... But then 
see above for the right solution.


Matt.

spool files not temp?

[Matt Sergeant]

Hi,

Can anyone remember the reason that the spool files aren't proper temp 
files (deleted upon open)? We often end up with a hard restart of qpsmtpd 
and having these files left around is annoying...


Matt.

Re: I am probably doing something wrong

[Matt Sergeant]

Steve wrote:
  
  Note that qpsmtpd probably should reject the message as invalid format. If

  it doesn't do that in the core, at least a standard plugin should do that
  validation.

 

The message is not rejected. Anyway... what plugin is doing the validation? Can 
you point me to the plugin doing such a validation?


There isn't one. Charlie is saying there should be such a plugin.

Though honestly it should probably be in core not a plugin.

Matt.

Re: New plugin: smtptls-forward

[Matt Sergeant]

On Tue, 30 Mar 2010, Charlie Brady wrote:


On Thu, 25 Mar 2010, Jason Mills wrote:


I wrote this plugin to help me with my local debugging.
Basically a heavily modified version of smtp-foward.


I'd recommend you search the archives and find some earlier comments by me 
about smtp-forward. It really should be rewritten to hook into more than just 
the queue hook, and pass sender and recipient addresses to the backend as 
they are provided by the connecting client, and relay responses back to the 
client.


I think some people might not want that, as they want to reduce the load 
on their main SMTP server... I agree it should be an option though. 
Perhaps we need queue/smtp-passthru.


Matt.

ParaDNS 2.0 - now does dns0x20

[Matt Sergeant]

dns0x20 protects against things like the Kaminsky DNS attack by vastly 
increasing the size of the keyspace for DNS requests.


It's enabled by default in ParaDNS 2.0. You can disable with an env var.

Matt.

Re: Release soon

[Matt Sergeant]

On Fri, 12 Feb 2010, Ask Bjørn Hansen wrote:


Hi everyone,

I'm going to make a release soon, so if you have patches that aren't merged 
into my branch yet that you think should be, be sure to speak up!

I just merged the RPM packaging stuff Peter Holzer and Robin Bowes made (over 
the last 5 years; don't say anything is being rushed around here!)  :-)

The shortlog since the last release (so far) is below.


I have the AnyEvent version of Qpsmtpd working if people are interested in 
getting it added to core... It's slower than the Danga::Socket version 
though.


Matt.

Re: Founding a Perlmongers group

[Matt Sergeant]

James Laver wrote:

As I shall shortly be leaving london for somewhere with no PM group, I
thought it might be nice to create one.

How do I go about it?
   


http://www.pm.org/start/index.html

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: 2 depend or not 2 depend

[Matt Sergeant]

Nicholas Clark wrote:

It does if you have a second machine to test on.
It doesn't if you have a shared development server, and the installed packages
are common to all developers.


Then the owners of those boxes need to learn about xen. And fast.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: 2 depend or not 2 depend

[Matt Sergeant]

Ash Berlin wrote:

  Use your OS's package management system.
 


Which is pretty much guaranteed to not have the exact versions they currently 
have installed if they've been using `cpan` et al. to install it .


I don't mean get them from the OS distributor. I mean build RPMs (or 
debs or whatever) and maintain them separately, including an SRPM 
archive if you ever need to rebuild stuff.


Use a tool, like cpan2rpm.

Matt.


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Bug#567364: Copyright and license for Perl module XML::Filter::XSLT

[Matt Sergeant]

I can confirm the license. If I ever get to updating the module I'll add 
the claim.


Ansgar Burchardt wrote:

Hi,

the Debian package for XML::Filter::XSLT claims that the module is
distributed under the same terms as perl (ie. Artistic License and
General Public License), but this is not documented anywhere else in the
distribution.

As you are listed as the author for the module, could you please clarify
under which terms the module is distributed?  It would be nice if you
could include years for the copyright.

A reply like

   The module ... is free software; you can redistribute it and/or modify
   it under the same terms as Perl itself.

   It is copyrightyear  by ...

would be enough.

Thanks,
Ansgar

PS: Please keep 567...@bugs.debian.org CCed in your reply.

   




--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#567364: Copyright and license for Perl module XML::Filter::XSLT

[Matt Sergeant]

I can confirm the license. If I ever get to updating the module I'll add 
the claim.


Ansgar Burchardt wrote:

Hi,

the Debian package for XML::Filter::XSLT claims that the module is
distributed under the same terms as perl (ie. Artistic License and
General Public License), but this is not documented anywhere else in the
distribution.

As you are listed as the author for the module, could you please clarify
under which terms the module is distributed?  It would be nice if you
could include years for the copyright.

A reply like

   The module ... is free software; you can redistribute it and/or modify
   it under the same terms as Perl itself.

   It is copyrightyear  by ...

would be enough.

Thanks,
Ansgar

PS: Please keep 567...@bugs.debian.org CCed in your reply.

   




--
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: badmailfrom per domain

[Matt Sergeant]

Johan Almqvist wrote:

On 21. jan. 2010, at 16.57, Christian Herndler wrote:
   

I get a lot of spam where the sender uses a whole /24 Subnet as mail
relay, the helo uses the pattern

mx{last-octet-of-ip}.domainname.net

so it could be blocked using the check_spamhelo plugin, but to do that
this plugin would need a small change as it only works with hostnames.
 


I've had a similar problem at the MAIL FROM level (right-hand side VERP, you 
could say) and fixed this with this plugin:

http://github.com/tyskjohan/qpsmtpd/blob/master/plugins/check_badmailfrom_patterns

   

Also consider using Enemies List on the HELO. It's very effective.

Re: Domain acquisition

[Matt Sergeant]

Jurgen Pletinckx wrote:

Hm. But that really only holds for domains you're actually using, or have
plans for, right? Can I actually find out which other domains the
proprietors hold? A reverse whois, so to say.
There are some services which can do this - they do it by downloading 
the .com zone every day and linking back to the nameservers. Doesn't 
always work (e.g. when using godaddy's or some other public DNS servers).


I might have an account on such a service if you need help.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

ParaDNS 1.9 out

[Matt Sergeant]

This fixes a major bug when under high load causing overly quick timeouts 
for some requests. Recommended upgrade.


(only used by those using -async)

Matt.

Re: Perl Christmas Quiz 2009

[Matt Sergeant]

On Mon, 30 Nov 2009 18:24:12 +, Chris Jack wrote:
 1) Without running it to check, what does the following program output?
 
 
 my %a = (3,2,1,0);
 
 
 for my $b (sort values %a) {
 $b += 4;
 }
 
 
 print $a{1} . \n;

Bizarrely enough, on both my Snow Leopard machines (default perl 
install) this outputs: 4D

Nice bug there.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Perl Christmas Quiz 2009

[Matt Sergeant]

On Tue, 1 Dec 2009 11:23:09 -0500, Matt Sergeant wrote:
 On Mon, 30 Nov 2009 18:24:12 +, Chris Jack wrote:
 1) Without running it to check, what does the following program output?
 
 
 my %a = (3,2,1,0);
 
 
 for my $b (sort values %a) {
 $b += 4;
 }
 
 
 print $a{1} . \n;
 
 Bizarrely enough, on both my Snow Leopard machines (default perl 
 install) this outputs: 4D
 
 Nice bug there.

Ah. It's the D from ^D. Shitty terminal.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: Perl Christmas Quiz 2009

[Matt Sergeant]

On Tue, 01 Dec 2009 08:47:55 -0800, Randal L. Schwartz wrote:
 Lemme guess.  You did this:
 
 $ perl
 ... type program in here ...
 ^D (control D)
 
 The D is from your control D.
 
 Common misconception.

Yes. Though oddly enough it doesn't show up in the same terminal when 
ssh'd into a Linux box. I'd like to know the reason why that is.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: AnyEvent mode?

[Matt Sergeant]

OK, here's the files as they currently stand. The big note on this is that 
I have done very little testing, and most importantly I have NOT updated 
any of the plugins (see the async/* dir for those that need re-written to 
use AnyEvent, mostly to use AnyEvent::DNS. Also the tls plugin would 
need updated/rewritten, though it's probably a lot simpler with AnyEvent).


I'll be trying to do performance testing tomorrow (comparing with 
qpsmtpd-async, NOT any of the other models).


Matt.#!/usr/bin/perl

use lib ./lib;
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}

# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;

use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib $FindBin::Bin/lib;
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;

$|++;

$SIG{'PIPE'} = IGNORE;  # handled manually

$DEBUG  = 0;

my $PORT= 2525;
my $LOCALADDR   = '0.0.0.0';
my $PROCS   = 1;
my $USER= (getpwuid $)[0]; # user to suid to
   $USER= smtpd if $USER eq root;
my $PAUSED  = 0;
my $NUMACCEPT   = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH;   # daemonize on startup

# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX = 1000;

sub reset_num_accept {
$NUMACCEPT = 20;
}

sub help {
print EOT;
Usage:
qpsmtpd [OPTIONS]

Options:
 -l, --listen-address addr : listen on a specific address; default 0.0.0.0
 -p, --port P  : listen on a specific port; default 2525
 -u, --user U  : run as a particular user; defualt 'smtpd'
 -j, --procs J : spawn J processes; default 1
 -d, --detach  : detach from controlling terminal (daemonize)
 --pid-file P  : print main servers PID to file P
 
 -h, --help: this page
EOT
exit(0);
}

GetOptions(
'p|port=i'  = \$PORT,
'l|listen-address=s'= \$LOCALADDR,
'j|procs=i' = \$PROCS,
'v|verbose+'  = \$DEBUG,
'u|user=s'  = \$USER,
'pid-file=s'= \$PID_FILE,
'd|detach'  = \$DETACH,
'h|help'= \help,
) || help();

# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { help }

use constant READY  = 1;
use constant ACCEPTING  = 2;
use constant RESTARTING = 999;

my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;

if ($PID_FILE  -r $PID_FILE) {
open PID, $PID_FILE
or die open_pidfile $PID_FILE: $!\n;
my $running_pid = PID || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die Found an already running qpsmtpd with pid $running_pid.\n;
}
}
close(PID);
}

run_as_server();

exit(0);

sub spawn_child {
my ($plugin_loader) = @_;

my $pid = fork;

if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die Unable to fork;
}

$plugin_loader-run_hooks('post-fork');

accept_loop();
}

sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print Connect?\n;
while ($SERVER  (my $peer = accept my $fh, $SERVER)) {
 fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside 
world does not

 my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
 my $qp = Qpsmtpd::AnyEvent-new($fh, format_address($host), 
$service);
 $qp-process_line(Connect);
  }
};

AnyEvent-condvar-wait;
exit;
}

sub sig_hup {
kill 1, keys %childstatus;
}

sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG))  0) {
if (!defined $childstatus{$child}) {
next;
}

last unless $child  0;
print SIGCHLD: child $child died\n;
delete $childstatus{$child};
$CURRENT_PROCS--;
}

$SIG{CHLD} = \sig_chld;
}

sub HUNTSMAN {
$SIG{CHLD} = 'DEFAULT';
kill 'INT' = keys %childstatus;
if ($PID_FILE  -e $PID_FILE) {
unlink $PID_FILE or ::log(LOGERROR, unlink: $PID_FILE: $!);
}
exit(0);
}

sub _connect_sock {
my $ipn = parse_address($LOCALADDR) or die cannot parse '$LOCALADDR' as 
host address;

my $af = address_family $ipn;

socket(my $sock, $af, SOCK_STREAM, 0) or die socket: $!;
setsockopt($sock, SOL_SOCKET, SO_REUSEADDR, 1)
or die tcp_server/so_reuseaddr: $!;

bind $sock, 

AnyEvent mode?

[Matt Sergeant]

Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written 
(mostly hacked right now, but should work).


In theory it might be faster than the Danga::Socket based one, and 
AnyEvent seems to receive regular updates more than Danga::Socket these 
days.


Matt.

Re: AnyEvent mode?

[Matt Sergeant]

On Mon, 23 Nov 2009, Ask Bjørn Hansen wrote:



On Nov 23, 2009, at 11:41, Matt Sergeant wrote:


Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written 
(mostly hacked right now, but should work).

In theory it might be faster than the Danga::Socket based one, and AnyEvent 
seems to receive regular updates more than Danga::Socket these days.


There are also more other stuff available with AnyEvent -- I think it'd be 
cool!


OK. How do I get this to you? It's basically just two new files, no 
patches to anything.


Matt.

Re: [sqlite] Get Max Samples

[Matt Sergeant]

On Tue, 10 Nov 2009 15:28:30 -0500, Pavel Ivanov wrote:
> You're right about max() and group_concat() will not help you either.
> You need something like this:
> 
> select max(cnt)
> from (select count(*) as cnt from table_name group by SampleNum)

That'll give you the count of the largest set. But not the actual 
value. For that you need to combine it with a HAVING clause. But I'll 
leave that as an exercise :-)

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Get Max Samples

[Matt Sergeant]

On Tue, 10 Nov 2009 15:28:30 -0500, Pavel Ivanov wrote:
 You're right about max() and group_concat() will not help you either.
 You need something like this:
 
 select max(cnt)
 from (select count(*) as cnt from table_name group by SampleNum)

That'll give you the count of the largest set. But not the actual 
value. For that you need to combine it with a HAVING clause. But I'll 
leave that as an exercise :-)

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: Language detection?

[Matt Sergeant]

On Sat, 7 Nov 2009, Johan Almqvist wrote:


Hi

On 5. nov. 2009, at 16.33, Matt Sergeant wrote:

Anyone got a language detection plugin? I get a lot of spam slipping 
through in spanish, and I'd like to just whack it on the head.


There's 
http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Plugin_TextCat.html 
but not sure if that's what you want.


Yeah there's plenty of ways to do it. Textcat being one of them. Google 
also has an API for language detection which is pretty good. I just 
wondered if anyone had anything pre-rolled for qpsmtpd.


Matt.

Re: [Bug 6155] generate new scores for 3.3.0 release

[Matt Sergeant]

On Thu, 22 Oct 2009 09:34:13 -0700, Michael Peddemors wrote:
 I am curious to the large HAM rate..  Again, I think the testing of 
 this rule 
 against a corpus might be affecting this.. 

I tend to agree. AOL announced wholesale blocking of anyone with 
NXDOMAIN rDNS a few years back now, and that caused big changes in 
people thinking it was OK to mail from an IP with NXDOMAIN rDNS.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: keyboards/RSI/switching costs (was Looking for a secondhand Datahand Pro II)

[Matt Sergeant]

On Wed, 21 Oct 2009 13:05:28 +0100, James Laver wrote:
 On Wed, Oct 21, 2009 at 12:31 PM, Chris Jack chris_j...@msn.com wrote:
 Before you switch keyboards, I think there is an important question 
 about how often you are obliged to use a standard qwerty keyboard. I 
 worked all over Europe for a bit using a large number of the 
 European variations on qwerty (y and z switched for instance and 
 punctuation in unusual places). I found the constant switching meant 
 I was slower on all keyboards - but maybe it was worse because the 
 keyboards were kind of the same. Maybe it's not such a problem if 
 you switch between, say, qwerty and colemak.
 
 A friend of mine in Canada tried it for a few weeks at work only (I
 figured there was no chance of losing productivity at work) and used
 qwerty at home and seemed to do fine with switching. No subtle
 differences, it's a whole different mode of typing.

Problem then comes with people who need to help you on your computer. I 
often help a tester here who has a Natural split keyboard, and find 
it tough, but doable (I used to use a natural years ago, the problem is 
using a Natural from a sideways position or standing position while at 
$co-worker's desk). I can't imagine any way of coping if he had a 
Dvorak layout.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: keyboards/RSI/switching costs (was Looking for a secondhand Datahand Pro II)

[Matt Sergeant]

On Wed, 21 Oct 2009 10:32:46 -0400, jesse wrote:
 Problem then comes with people who need to help you on your computer. I 
 often help a tester here who has a Natural split keyboard, and find 
 it tough, but doable (I used to use a natural years ago, the problem is 
 using a Natural from a sideways position or standing position while at 
 $co-worker's desk). I can't imagine any way of coping if he had a 
 Dvorak layout.
 
 When my friend Adam was switching over to Dvorak many years ago, he
 implemented two small shell scripts to toggle layout: asdf and aoeu 
 
 It helped keep the problem Matt mentions in check.

True-ish. If you occasionally glance at the keys it really screws you 
over though :)

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: keyboards/RSI/switching costs (was Looking for a secondhand Datahand Pro II)

[Matt Sergeant]

On Wed, 21 Oct 2009 16:32:15 +0100, James Laver wrote:
 On Wed, Oct 21, 2009 at 4:02 PM, Matt Sergeant
 mserge...@messagelabs.com wrote:
 
 True-ish. If you occasionally glance at the keys it really screws you
 over though :)
 
 Well unless you're buying labels to stick on the keys you aren't going
 to be able to look down on Dvorak, and I'd hope that by now you've
 muscle-memorised QWERTY.

On my own keyboard, yes. The point was on someone else's keyboard.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: [PATCH] Log even when we aren't in a transaction

[Matt Sergeant]

On Mon, 24 Aug 2009 09:55:52 -0500, Jared Johnson wrote:
 On 08/23/2009 04:03 PM, Matt Sergeant wrote:
 On Thu, 20 Aug 2009 10:18:39 -0500, Jared Johnson wrote:
 It looks like logging/file doesn't like the empty hashref returned by
 Qpsmtpd::transaction().
 
 I never understood why it did that. Any reason it can't return either
 undef or (preferably) a new Transaction object?
 
 I don't really understand it either, and I fear that which I don't 
 understand, so I worry about taking it out and breaking some hackage 
 that depends no it :)

Yeah agreed. I think it'd be worth checking with Ask why he made it 
this way.

Re: [PATCH] Log even when we aren't in a transaction

[Matt Sergeant]

On Thu, 20 Aug 2009 10:18:39 -0500, Jared Johnson wrote:
 It looks like logging/file doesn't like the empty hashref returned by 
 Qpsmtpd::transaction().

I never understood why it did that. Any reason it can't return either 
undef or (preferably) a new Transaction object?

Re: [sqlite] Does PRAGMA synchronous=OFF ensure that no synching is done for the entire session?

[Matt Sergeant]

On Mon, 17 Aug 2009 10:47:23 -0400, Angus March wrote:
>> Because yes, that's what synchronous=OFF means. It stops SQLite from 
>> issuing fflush calls (effectively).
>>   
> Right, and this is implied by the documentation, but I was concerned
> that the documentation might be playing fast and loose, saying that
> fflush (or fsync, or fdatasync) won't be called, when it really means
> that it won't be called during any call to step() or finalize(), while
> it would be called when the session is closed. I wasn't sure, so I
> thought I'd ask, because it'll matter to my app.

Kernels will fflush when a file handle is closed, which will happen 
when you close the database handle.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does PRAGMA synchronous=OFF ensure that no synching is done for the entire session?

[Matt Sergeant]

On Mon, 17 Aug 2009 10:47:23 -0400, Angus March wrote:
 Because yes, that's what synchronous=OFF means. It stops SQLite from 
 issuing fflush calls (effectively).
   
 Right, and this is implied by the documentation, but I was concerned
 that the documentation might be playing fast and loose, saying that
 fflush (or fsync, or fdatasync) won't be called, when it really means
 that it won't be called during any call to step() or finalize(), while
 it would be called when the session is closed. I wasn't sure, so I
 thought I'd ask, because it'll matter to my app.

Kernels will fflush when a file handle is closed, which will happen 
when you close the database handle.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does PRAGMA synchronous=OFF ensure that no synching is done for the entire session?

[Matt Sergeant]

On Fri, 14 Aug 2009 12:33:30 -0400, Angus March wrote:
> I want my INSERT done right away, I just don't want it to be flushed
> from the filesystem's write-behind cache until the kernel decides, not
> when SQLite decides.

Did you mean you do "want it to be flushed from the filesystem's 
write-behind cache when the kernel decides (rather than when SQLite 
decides)"?

Because yes, that's what synchronous=OFF means. It stops SQLite from 
issuing fflush calls (effectively).

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does PRAGMA synchronous=OFF ensure that no synching is done for the entire session?

[Matt Sergeant]

On Fri, 14 Aug 2009 12:33:30 -0400, Angus March wrote:
 I want my INSERT done right away, I just don't want it to be flushed
 from the filesystem's write-behind cache until the kernel decides, not
 when SQLite decides.

Did you mean you do want it to be flushed from the filesystem's 
write-behind cache when the kernel decides (rather than when SQLite 
decides)?

Because yes, that's what synchronous=OFF means. It stops SQLite from 
issuing fflush calls (effectively).

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: testing Malware Patrol rules?

[Matt Sergeant]

On Fri, 24 Jul 2009 16:09:46 +0300, Henrik Krohns wrote:
 On Fri, Jul 24, 2009 at 09:45:42AM +, Justin Mason wrote:
 hi Andre --
 
 A SpamAssassin user mentioned this ruleset today:
 
   http://malware.hiperlinks.com.br/cgi/submit?action=list_sa
 
 it looks good!  Would you mind if I added a copy of that to our rule-QA
 system (http://ruleqa.spamassassin.org/), primarily to determine false
 positive rate?
 
 If that goes well, btw, a possibility would be that I could generate a
 SpamAssassin rule updates channel for you, similar to how the sought
 ruleset works: http://wiki.apache.org/spamassassin/SoughtRules .  Let me
 know if you're interested in that.
 
 I would add \b or so in front of the sigs..
 
 For example, /zief\.pl\//i should be /\bzief\.pl\//i. Unbounded short
 domains like that have chances of FPs.

Plus they should be URI rules, otherwise you're just re-scanning the 
entire body.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__

Re: [sqlite] SQLite3 immune from injection attacks if file is readonly?

[Matt Sergeant]

On Sat, 18 Jul 2009 10:17:14 -0700, Kelly Jones wrote:
> On a website, I want to take a user's query "as is", save it to a
> userquery.txt, and then do:
> 
> sqlite3 /path/to/mydb < userquery.txt
> 
> where /path/to/mydb is a *read-only* file.
> 
> Is there *any* risk of an injection attack here?

Yes. Massively. You need to read up on the fundamentals of SQL 
injection, and use a proper API for accessing the DB that allows you to 
use bind variables.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite3 immune from injection attacks if file is readonly?

[Matt Sergeant]

On Sat, 18 Jul 2009 10:17:14 -0700, Kelly Jones wrote:
 On a website, I want to take a user's query as is, save it to a
 userquery.txt, and then do:
 
 sqlite3 /path/to/mydb  userquery.txt
 
 where /path/to/mydb is a *read-only* file.
 
 Is there *any* risk of an injection attack here?

Yes. Massively. You need to read up on the fundamentals of SQL 
injection, and use a proper API for accessing the DB that allows you to 
use bind variables.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Performance against huge datasets

[Matt Sergeant]

On Tue, 23 Jun 2009 22:01:26 +0200, Misza wrote:
> I wonder if anyone used SQLite extensively with big datasets and could
> provide some insight into performance?
> In a nutshell, I am writing an ETL framework and need a good (read:
> performing) engine for the "T"ransform part.
> I suppose I could use flat files for that, but I'd like to have some SQL
> capabilities at my disposal, which is why I'm poking around file-based,
> serverless engines.
> The question is, how does SQLite perform when faced with huge datasets,
> where "huge" means 10s of gigabytes in size (typical for a Data
> Warehouse's staging area)?
> Most common operations (after unload) would include multi-table joins
> (mostly merge joins), field transformations (contatenation, casting) and
> record filtering.

You might want to consider Hive.

http://www.facebook.com/note.php?note_id=89508453919

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Performance against huge datasets

[Matt Sergeant]

On Tue, 23 Jun 2009 22:01:26 +0200, Misza wrote:
 I wonder if anyone used SQLite extensively with big datasets and could
 provide some insight into performance?
 In a nutshell, I am writing an ETL framework and need a good (read:
 performing) engine for the Transform part.
 I suppose I could use flat files for that, but I'd like to have some SQL
 capabilities at my disposal, which is why I'm poking around file-based,
 serverless engines.
 The question is, how does SQLite perform when faced with huge datasets,
 where huge means 10s of gigabytes in size (typical for a Data
 Warehouse's staging area)?
 Most common operations (after unload) would include multi-table joins
 (mostly merge joins), field transformations (contatenation, casting) and
 record filtering.

You might want to consider Hive.

http://www.facebook.com/note.php?note_id=89508453919

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: qpsmtpd-async authenticated relaying direction request

[Matt Sergeant]

On Sat, 30 May 2009, Baltasar Cevc wrote:


On May 28, 2009, at 4:37 PM, Matt Sergeant wrote:

Seems a little fragile. There aren't many bounces that quote all

headers. You'd be better off just rejecting all bounces in qpsmtpd,
then you only see legit bounces where the remote end issued an
immediate 5xx to your Exim's outgoing mail. Of course when I suggest
that (here and on other lists) mail admins tend to freak out a bit. But
I'm of the opinion that I've barely ever seen a useful full bounce in
years. So my qpsmtpd runs a no_bounces plugin, which I believe I've
posted here before.


While I would not freak out, it is surely not RFC conformant. And, even
worse, it is illegal in some legislations (I can tell for Germany):
suppressing mails (thus also failure notices that made sure the sender
knows a mail has not arrive) is punishable under the German Criminal
Code.


That's a major misunderstanding of both the RFCs and German law.


However, I think everybody should descide himself/herself.


Indeed.


I notice 'broken' systems that will send bounces instead of rejecting
mail from time to time...


I'm not suggesting there aren't. But as far as I can tell, there's 3 
options:


1) accept all bounces, and get flooded with invalid bounces when spammers 
spoof you.


2) implement BATV and the flaws it has.

3) reject all SMTP level bounces and the flaws it has.

All have problems. You just pick your acceptable failures.

Matt.

Re: qpsmtpd-async authenticated relaying direction request

[Matt Sergeant]

On Wed, 27 May 2009 09:18:32 -0500, David Favor wrote:
 I'm currently running qpsmtpd-async.
 
 I host many domains and I'd like to protect them all
 against backscatter using something like this:
 
 http://psg.com/~brian/software/authbounce/configure-authbounce.txt
 
 to add a bounce key to each outgoing message of the form:
 
 X-bounce-key: $mx-$number;$sender;$timestamp;$key
 
 This requires all mail sent by every user to go through
 qpsmtpd + exim on my local machine.

Seems a little fragile. There aren't many bounces that quote all 
headers. You'd be better off just rejecting all bounces in qpsmtpd, 
then you only see legit bounces where the remote end issued an 
immediate 5xx to your Exim's outgoing mail. Of course when I suggest 
that (here and on other lists) mail admins tend to freak out a bit. But 
I'm of the opinion that I've barely ever seen a useful full bounce in 
years. So my qpsmtpd runs a no_bounces plugin, which I believe I've 
posted here before.

Matt.

Re: qpsmtpd-async authenticated relaying direction request

[Matt Sergeant]

On Thu, 28 May 2009 12:04:27 -0400 (EDT), Charlie Brady wrote:
 
 On Thu, 28 May 2009, Matt Sergeant wrote:
 
 years. So my qpsmtpd runs a no_bounces plugin, which I believe I've
 posted here before.
 
 Google seems not to have heard of it.

Ah. OK. It basically just does this:

  if ($transaction-sender-format eq 
  or
  $transaction-sender-format =~ /MAILER-DAEMON/i
  or
  $transaction-sender-format =~ /postmaster\@/i
  )
  {
unless (grep { /(bounce-|-return-)/ } map { $_-address} 
$transaction-recip
ients) {
  return DENY, Mail from  not accepted here;
}
  }

Re: Feature request to disable CONTROL_PORT

[Matt Sergeant]

On Wed, 27 May 2009, David Favor wrote:


Having qpsmtpd listen on an additional control port
creates serious complexity when running multiple
copies of qpsmtpd, as each copy has to somehow figure
out which control port to use, hope it's free and then
connect.

I usually just strip this code out of qpsmtpd or
comment out the initial connection.

A great feature to add is a simple command line option
to turn this off.


Yeah, this is only on -async anyway. I'm wondering if we should just dump 
the feature, since it doesn't work when having multiple async children.


Matt.

Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81

[Matt Sergeant]

On Thu, 21 May 2009, Devin Carraway wrote:


On Wed, May 20, 2009 at 09:40:21PM -0400, Charlie Brady wrote:

I think the -T *should* be there on the command line, but there are some
bugs in qpsmtpd and/or your plugins which need to be fixed before it will
work.


forkserver has used -T since 29ac2860, back in 2004.  Obviously prefork is
newer and has seen less testing, but most of the module code and plugins have
seen plenty of taint-checked use.


I guess this raises a question: The return values from config() are 
tainted. Should we de-taint them?


Matt.

Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81

[Matt Sergeant]

On Wed, 20 May 2009, J wrote:


I compared the run file with other run files (i.e. djbdns and qmail) and I
think the problem is with the trailing ' \' on the 2nd line (the first
exec).


Indeed. That shouldn't be there.


When I remove that (and installed a missing Math::BigInt package from
CPAN), everything loads, but complains about an insecure dependency on
line 416 in setpriority (in qpsmtpd-prefork). (And the prefork processes
keep showing up in the process table, and dying.) I expect that this was
tested before being released, so there is probably still something wrong
with my setup :-(


I think the -T shouldn't be there on the command line. Though it is 
probably a bug, I'm guessing we don't test with taint on.


Matt.

Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81

[Matt Sergeant]

On Wed, 20 May 2009, Charlie Brady wrote:


Though it is probably a bug, I'm guessing we don't test with taint on.


Perl taint mode is an underutilised gem.


It is, but it's also buggy and annoying.

(there's a completely ignored bug in perl with -T and hash keys which I 
filed months ago)


Matt.

Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81

[Matt Sergeant]

On Wed, 20 May 2009, David Nicol wrote:


On Wed, May 20, 2009 at 10:28 PM, Matt Sergeant m...@sergeant.org wrote:


(there's a completely ignored bug in perl with -T and hash keys which I
filed months ago)



that hash keys are never tainted is documented, if that's your bug.  It
allows for a quick and dirty

  sub detaint($){ [ keys %{{ $_[0] = 1 }} ] - [0] }


Nope. #56842. Very annoying.

Matt.

Re: Tokyo Cabinet as a BayesStore

[Matt Sergeant]

On Thu, 23 Apr 2009 10:32:13 +, Justin Mason wrote:
 
http://anyall.org/blog/2009/04/performance-comparison-keyvalue-stores-for-language-model-counts/
 
 highlight: a Tokyo Cabinet hashtable performed at 1400 ops/sec compared to
 BerkeleyDB's 340 (via python bindings), over 4 times faster.  There's been
 a lot of good press about it possibly a candidate for a future plugin?

The times look really bizarre to me.

An in memory store can only do 2700 tweets/sec (whatever that 
means)??? That's INCREDIBLY low.

I suspect BerkelyDB there is at about as fast as you might get without 
turning off fsync to disk. Tokyo Cabinet is probably faster because it 
doesn't fsync. I imagine that's about all there is to it.

Would love to be proven wrong though.

Matt.

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__