Re: filter oddities

2020-01-29 Thread Edgar Pettijohn 




On 01/25/20 16:14, Edgar Pettijohn wrote:



On 01/25/20 14:48, Edgar Pettijohn wrote:



On 01/25/20 14:24, gil...@poolp.org wrote:
January 25, 2020 9:21 PM, "Edgar Pettijohn" 
 wrote:



On 01/25/20 14:20, gil...@poolp.org wrote:

January 25, 2020 8:50 PM, "Edgar Pettijohn" 
 wrote:


I haven't seen any mention of this, but for some reason in my 
limited "testing" of filters I have
to use \r\n in my responses to smtpd. Is this normal? Doesn't 
seem to be documented and what

filters I've looked at don't appear to be using \r\n.

Indeed, you must certainly NOT use "\r\n" in filters.

For example without ORS = "\r\n" the following script will cause 
smtpd to basically just hang.
There is no errors reported, but when I attempt to telnet 
localhost 25 the daemon doesn't greet me.

After adding ORS = "\r\n" everything works as expected.
I'm not familiar with awk beyond very basic uses, could this be 
due to some
flushing not happening by default and which gets triggere with 
"\r\n" ?

Adding:

setvbuf(stdout, NULL, _IONBUF, 0);

to awk's main() fixes it on my laptop. Still don't understand how my 
laptop is apparently the only device effected...


At least I can continue testing on my laptop anyway.

Edgar

Re: filter oddities

2020-01-26 Thread Edgar Pettijohn 




On 01/26/20 01:01, Edgar Pettijohn wrote:

On Jan 26, 2020 12:11 AM, Martijn van Duren  
wrote:

On 1/25/20 8:50 PM, Edgar Pettijohn wrote:

I haven't seen any mention of this, but for some reason in my limited
"testing" of filters I have to use \r\n in my responses to smtpd. Is
this normal? Doesn't seem to be documented and what filters I've looked
at don't appear to be using \r\n.


For example without ORS = "\r\n" the following script will cause smtpd
to basically just hang. There is no errors reported, but when I attempt
to telnet localhost 25 the daemon doesn't greet me. After adding ORS =
"\r\n" everything works as expected.

#!/usr/bin/awk -f
BEGIN {
       ARGC = 0
       FS = "|"
       OFS = FS
       ORS = "\r\n"
}

"config|ready" == $0 {
       print "register|report|smtp-in|*"
       print "register|ready"
       print $0 >> "/tmp/report.txt"
       next
}

"config" == $1 {
       next
}

"report" == $1 {
       print $0 >> "/tmp/report.txt"
       next
}



I tried both your awk script without ORS and perl script a
OpenBSD-current machine and both work as expected. So either there's
something weird in -portable or there's something weird with your
setup.

I don't have a NetBSD machine at hand, so could you provide me shell-
access to a machine that can reproduce this problem? I can probably
take a look at it in the coming week.

Will do. I'll spin one up and get it ready for you tomorrow hopefully.

Thanks,

Edgar


Must be a local issue to my laptop. I spun up a server at vultr and got 
everything ready. Then tested and it works both with and without the 
/r/n. I don't understand what the difference is between my laptop and 
the server. Either way it appears to be resolved.


Thanks,

Edgar

Re: filter oddities

2020-01-25 Thread Edgar Pettijohn 




On 01/25/20 14:48, Edgar Pettijohn wrote:



On 01/25/20 14:24, gil...@poolp.org wrote:
January 25, 2020 9:21 PM, "Edgar Pettijohn"  
wrote:



On 01/25/20 14:20, gil...@poolp.org wrote:

January 25, 2020 8:50 PM, "Edgar Pettijohn" 
 wrote:


I haven't seen any mention of this, but for some reason in my 
limited "testing" of filters I have
to use \r\n in my responses to smtpd. Is this normal? Doesn't seem 
to be documented and what

filters I've looked at don't appear to be using \r\n.

Indeed, you must certainly NOT use "\r\n" in filters.

For example without ORS = "\r\n" the following script will cause 
smtpd to basically just hang.
There is no errors reported, but when I attempt to telnet 
localhost 25 the daemon doesn't greet me.

After adding ORS = "\r\n" everything works as expected.
I'm not familiar with awk beyond very basic uses, could this be due 
to some
flushing not happening by default and which gets triggere with 
"\r\n" ?


This is another awk filter which doesn't use ORS:

https://github.com/jirutka/opensmtpd-filter-rewrite-from/blob/master/filter-rewrite-from 



so I'm not sure why yours block but the solution is not with "\r\n" 
for sure

I get the same with filter-rewrite-from.

Can you run filter traces while you reproduce ?


Doesn't seem to do much.

laptop$ telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.

laptop$ sudo smtpd -dv -T filters
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
info: OpenSMTPD 6.6.1-portable starting
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ca-tree
debug: init ca-tree
debug: init ssl-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
debug: using "ram" stat backend
setup_peer: control -> klondike[6603] fd=5
setup_peer: lookup -> control[18573] fd=5
setup_peer: control -> lookup[27107] fd=6
setup_peer: lookup -> pony express[16452] fd=6
setup_peer: control -> pony express[16452] fd=7
setup_peer: lookup -> queue[4582] fd=7
setup_peer: control -> queue[4582] fd=8
setup_peer: control -> scheduler[28554] fd=9
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: klondike -> control[18573] fd=5
setup_peer: klondike -> pony express[16452] fd=6
debug: init ca-tree
setup_done: ca[6603] done
setup_proc: klondike done
setup_proc: control done
setup_done: control[18573] done
debug: init ssl-tree
setup_proc: lookup done
setup_done: lka[27107] done
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: pony express -> control[18573] fd=5
filters init type=proc, name=filter-perl, proc=filter-perl
setup_peer: pony express -> klondike[6603] fd=6
setup_peer: pony express -> lookup[27107] fd=7
setup_peer: pony express -> queue[4582] fd=8
setup_done: pony[16452] done
setup_proc: pony express done
debug: rsa_engine_init: using RSA privsep engine
debug: ecdsa_engine_init: using ECDSA privsep engine
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: scheduler -> control[18573] fd=5
setup_peer: queue -> control[18573] fd=5
setup_peer: scheduler -> queue[4582] fd=6
setup_peer: queue -> pony express[16452] fd=6
setup_peer: queue -> lookup[27107] fd=7
setup_peer: queue -> scheduler[28554] fd=8
setup_proc: queue done
setup_done: queue[4582] done
setup_proc: scheduler done
debug: bounce warning after 4h
setup_done: scheduler[28554] done
smtpd: setup done
debug: parent_send_config_ruleset: reloading
debug: parent_send_config: configuring pony process
debug: parent_send_config: configuring ca process
debug: init private ssl-tree
debug: smtp: listen on 127.0.0.1 port 25 flags 0x2400 pki "" ca ""
debug: smtp: listen on [::1] port 25 flags 0x2400 pki "" ca ""
debug: smtp: listen on [fe80::1%lo0] port 25 flags 0x2400 pki "" ca ""
debug: smtp: will accept at most 1697 clients
debug: queue: done loading queue into scheduler
debug: smtpd: scanning offline queue...
debug: smtpd: offline scanning done

I also have the same problem with perl:

#!/usr/pkg/bin/perl
#

open(my $fh, ">>", "/tmp/report.txt");

select($fh);
$|++;
select(stdout);
$|++;

while (my $line = <>) {
    if ($line =~ /config|ready/) {
    print "register|report|smtp-in|*\r\n";
    print "register|ready\r\n";
    #print "register|report|smtp-in|*";
    #print

Re: filter oddities

2020-01-25 Thread Edgar Pettijohn 




On 01/25/20 14:24, gil...@poolp.org wrote:

January 25, 2020 9:21 PM, "Edgar Pettijohn"  wrote:


On 01/25/20 14:20, gil...@poolp.org wrote:


January 25, 2020 8:50 PM, "Edgar Pettijohn"  wrote:


I haven't seen any mention of this, but for some reason in my limited "testing" 
of filters I have
to use \r\n in my responses to smtpd. Is this normal? Doesn't seem to be 
documented and what
filters I've looked at don't appear to be using \r\n.

Indeed, you must certainly NOT use "\r\n" in filters.


For example without ORS = "\r\n" the following script will cause smtpd to 
basically just hang.
There is no errors reported, but when I attempt to telnet localhost 25 the 
daemon doesn't greet me.
After adding ORS = "\r\n" everything works as expected.

I'm not familiar with awk beyond very basic uses, could this be due to some
flushing not happening by default and which gets triggere with "\r\n" ?

This is another awk filter which doesn't use ORS:

https://github.com/jirutka/opensmtpd-filter-rewrite-from/blob/master/filter-rewrite-from

so I'm not sure why yours block but the solution is not with "\r\n" for sure

I get the same with filter-rewrite-from.

Can you run filter traces while you reproduce ?


Doesn't seem to do much.

laptop$ telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.

laptop$ sudo smtpd -dv -T filters
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
info: OpenSMTPD 6.6.1-portable starting
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ca-tree
debug: init ca-tree
debug: init ssl-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
debug: using "ram" stat backend
setup_peer: control -> klondike[6603] fd=5
setup_peer: lookup -> control[18573] fd=5
setup_peer: control -> lookup[27107] fd=6
setup_peer: lookup -> pony express[16452] fd=6
setup_peer: control -> pony express[16452] fd=7
setup_peer: lookup -> queue[4582] fd=7
setup_peer: control -> queue[4582] fd=8
setup_peer: control -> scheduler[28554] fd=9
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: klondike -> control[18573] fd=5
setup_peer: klondike -> pony express[16452] fd=6
debug: init ca-tree
setup_done: ca[6603] done
setup_proc: klondike done
setup_proc: control done
setup_done: control[18573] done
debug: init ssl-tree
setup_proc: lookup done
setup_done: lka[27107] done
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: pony express -> control[18573] fd=5
filters init type=proc, name=filter-perl, proc=filter-perl
setup_peer: pony express -> klondike[6603] fd=6
setup_peer: pony express -> lookup[27107] fd=7
setup_peer: pony express -> queue[4582] fd=8
setup_done: pony[16452] done
setup_proc: pony express done
debug: rsa_engine_init: using RSA privsep engine
debug: ecdsa_engine_init: using ECDSA privsep engine
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: scheduler -> control[18573] fd=5
setup_peer: queue -> control[18573] fd=5
setup_peer: scheduler -> queue[4582] fd=6
setup_peer: queue -> pony express[16452] fd=6
setup_peer: queue -> lookup[27107] fd=7
setup_peer: queue -> scheduler[28554] fd=8
setup_proc: queue done
setup_done: queue[4582] done
setup_proc: scheduler done
debug: bounce warning after 4h
setup_done: scheduler[28554] done
smtpd: setup done
debug: parent_send_config_ruleset: reloading
debug: parent_send_config: configuring pony process
debug: parent_send_config: configuring ca process
debug: init private ssl-tree
debug: smtp: listen on 127.0.0.1 port 25 flags 0x2400 pki "" ca ""
debug: smtp: listen on [::1] port 25 flags 0x2400 pki "" ca ""
debug: smtp: listen on [fe80::1%lo0] port 25 flags 0x2400 pki "" ca ""
debug: smtp: will accept at most 1697 clients
debug: queue: done loading queue into scheduler
debug: smtpd: scanning offline queue...
debug: smtpd: offline scanning done

I also have the same problem with perl:

#!/usr/pkg/bin/perl
#

open(my $fh, ">>", "/tmp/report.txt");

select($fh);
$|++;
select(stdout);
$|++;

while (my $line = <>) {
    if ($line =~ /config|ready/) {
    print "register|report|smtp-in|*\r\n";
    print "register|ready\r\n";
    #print "register|report|smtp-in|*";
    #print "register|ready";
    }

    chomp $line;

  

Re: filter oddities

2020-01-25 Thread gilles 
January 25, 2020 9:21 PM, "Edgar Pettijohn"  wrote:

> On 01/25/20 14:20, gil...@poolp.org wrote:
> 
>> January 25, 2020 8:50 PM, "Edgar Pettijohn"  wrote:
>> 
>>> I haven't seen any mention of this, but for some reason in my limited 
>>> "testing" of filters I have
>>> to use \r\n in my responses to smtpd. Is this normal? Doesn't seem to be 
>>> documented and what
>>> filters I've looked at don't appear to be using \r\n.
>> 
>> Indeed, you must certainly NOT use "\r\n" in filters.
>> 
>>> For example without ORS = "\r\n" the following script will cause smtpd to 
>>> basically just hang.
>>> There is no errors reported, but when I attempt to telnet localhost 25 the 
>>> daemon doesn't greet me.
>>> After adding ORS = "\r\n" everything works as expected.
>> 
>> I'm not familiar with awk beyond very basic uses, could this be due to some
>> flushing not happening by default and which gets triggere with "\r\n" ?
>> 
>> This is another awk filter which doesn't use ORS:
>> 
>> https://github.com/jirutka/opensmtpd-filter-rewrite-from/blob/master/filter-rewrite-from
>> 
>> so I'm not sure why yours block but the solution is not with "\r\n" for sure
> 
> I get the same with filter-rewrite-from.

Can you run filter traces while you reproduce ?

Re: filter oddities

2020-01-25 Thread Edgar Pettijohn 




On 01/25/20 14:20, gil...@poolp.org wrote:

January 25, 2020 8:50 PM, "Edgar Pettijohn"  wrote:


I haven't seen any mention of this, but for some reason in my limited "testing" 
of filters I have
to use \r\n in my responses to smtpd. Is this normal? Doesn't seem to be 
documented and what
filters I've looked at don't appear to be using \r\n.


Indeed, you must certainly NOT use "\r\n" in filters.



For example without ORS = "\r\n" the following script will cause smtpd to 
basically just hang.
There is no errors reported, but when I attempt to telnet localhost 25 the 
daemon doesn't greet me.
After adding ORS = "\r\n" everything works as expected.


I'm not familiar with awk beyond very basic uses, could this be due to some
flushing not happening by default and which gets triggere with "\r\n" ?

This is another awk filter which doesn't use ORS:

https://github.com/jirutka/opensmtpd-filter-rewrite-from/blob/master/filter-rewrite-from

so I'm not sure why yours block but the solution is not with "\r\n" for sure

I get the same with filter-rewrite-from.

Re: filter oddities

2020-01-25 Thread gilles 
January 25, 2020 8:50 PM, "Edgar Pettijohn"  wrote:

> I haven't seen any mention of this, but for some reason in my limited 
> "testing" of filters I have
> to use \r\n in my responses to smtpd. Is this normal? Doesn't seem to be 
> documented and what
> filters I've looked at don't appear to be using \r\n.
> 

Indeed, you must certainly NOT use "\r\n" in filters.


> For example without ORS = "\r\n" the following script will cause smtpd to 
> basically just hang.
> There is no errors reported, but when I attempt to telnet localhost 25 the 
> daemon doesn't greet me.
> After adding ORS = "\r\n" everything works as expected.
> 

I'm not familiar with awk beyond very basic uses, could this be due to some
flushing not happening by default and which gets triggere with "\r\n" ?

This is another awk filter which doesn't use ORS:

https://github.com/jirutka/opensmtpd-filter-rewrite-from/blob/master/filter-rewrite-from

so I'm not sure why yours block but the solution is not with "\r\n" for sure