Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Robert Theisen 
Did you find anything of concern in the mimedefang process on your
host that was consuming 8GB RAM?
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Robert Theisen 
sendmail-milter-8.14.7-5.el7.x86_64

[ext-mx09] [10:31:02 PM]
[root@ext-mx09 tmp]# rpm -qa | grep milter
sendmail-milter-8.14.7-5.el7.x86_64
[ext-mx09] [10:31:10 PM]
[root@ext-mx09 tmp]# rpm -ql sendmail-milter
/usr/lib64/libmilter.so.1.0
/usr/lib64/libmilter.so.1.0.1
/usr/share/doc/sendmail-8.14.7/README.libmilter
/usr/share/doc/sendmail-milter-8.14.7
/usr/share/doc/sendmail-milter-8.14.7/LICENSE
[ext-mx09] [10:32:37 PM]
[root@ext-mx09 tmp]# ldd /usr/bin/mimedefang
linux-vdso.so.1 =>  (0x7ffc78ff7000)
libmilter.so.1.0 => /lib64/libmilter.so.1.0 (0x7ff833eb6000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7ff833c9a000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x7ff833a8)
libc.so.6 => /lib64/libc.so.6 (0x7ff8336bd000)
/lib64/ld-linux-x86-64.so.2 (0x7ff8340d)
[ext-mx09] [10:32:45 PM]
[root@ext-mx09 tmp]#

On Fri, Nov 10, 2017 at 4:09 PM, Dianne Skoll  wrote:
> On Fri, 10 Nov 2017 13:18:15 -0500
> Robert Theisen  wrote:
>
>> {
>> struct privdata *data;
>> char *val;
>> char buf[256];
>> if (*macro && *(macro+1)) {
>> /* Longer than 1 char -- use curlies */
>> snprintf(buf, sizeof(buf), "{%s}", macro);
>> val = smfi_getsymval(ctx, buf);
>> } else {
>> val = smfi_getsymval(ctx, macro);
>> }
>>  ...
>> }
>
>>
>> at snprintf(buf, sizeof(buf), "{%s}", macro);
>>
>> that buf[256] is not getting cleaned up even though it is a local
>> variable that is falling out of scope?
>
> Um what?  There's nothing to clean up.  CLeaning that up simply
> involves the C compiler incrementing the stack pointer by sizeof(buf);
> it's an automatic variable.
>
>> The strike against that theory would be that my memory dump shows that
>> the "i" and "j" variables are included in that memory chunk.  But the
>> single character i and j values are never copied to the local buf[256]
>> variable.
>
> If you're linking against a version of libmilter older than 8.14.4, the
> library has a memory leak in it that could be causing this.
>
> Regards,
>
> Dianne.
> ___
> NOTE: If there is a disclaimer or other legal boilerplate in the above
> message, it is NULL AND VOID.  You may ignore it.
>
> Visit http://www.mimedefang.org and http://www.roaringpenguin.com
> MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
> http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Dianne Skoll 
On Fri, 10 Nov 2017 13:18:15 -0500
Robert Theisen  wrote:

> {
> struct privdata *data;
> char *val;
> char buf[256];
> if (*macro && *(macro+1)) {
> /* Longer than 1 char -- use curlies */
> snprintf(buf, sizeof(buf), "{%s}", macro);
> val = smfi_getsymval(ctx, buf);
> } else {
> val = smfi_getsymval(ctx, macro);
> }
>  ...
> }

> 
> at snprintf(buf, sizeof(buf), "{%s}", macro);
> 
> that buf[256] is not getting cleaned up even though it is a local
> variable that is falling out of scope?

Um what?  There's nothing to clean up.  CLeaning that up simply
involves the C compiler incrementing the stack pointer by sizeof(buf);
it's an automatic variable.

> The strike against that theory would be that my memory dump shows that
> the "i" and "j" variables are included in that memory chunk.  But the
> single character i and j values are never copied to the local buf[256]
> variable.

If you're linking against a version of libmilter older than 8.14.4, the
library has a memory leak in it that could be causing this.

Regards,

Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Dianne Skoll 
Hi again,

What version of libmilter are you linking against?  There was a memory leak
in libmilter in versions older then 8.14.4.

https://groups.google.com/forum/#!topic/linux.debian.bugs.dist/I3QpUbpdwSM

Regards,

Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Robert Theisen 
Something that might help.  I noticed that the format of the string in
memory is ...

_ {auth_authen} {auth_author} {auth_ssf} {auth_type} {cert_issuer}
{cert_subject} {cipher} {cipher_bits} {daemon_name} {daemon_port} i
{if_addr} {if_name} j {mail_addr} {mail_host} {mail_mailer}
{tls_version} {verify} {rcpt_addr} {rcpt_host} {rcpt_mailer}


It leads with the _ and does have the curly braces around each item
longer than a single char.
It also terminates at {rcpt_mailer}

The curly braces are not added until the string is processed by
append_macro_value() and then immediately after append_macro_value()
runs, we add another value to dbuf called "client_port".  But client
port does not show up in the memory dump.


/* Write the standard macros */
macro = StandardSendmailMacros;
while (*macro) {
append_macro_value(, ctx, *macro);
macro++;
}

/* Fake client_port: We don't get the macro, but we have the connection
   info cached in our private data area. */
dbuf_putc(, '=');
append_percent_encoded(, "client_port");



Is it possible that during this function ...

static void
append_macro_value(dynamic_buffer *dbuf,
   SMFICTX *ctx,
   char *macro)
{
struct privdata *data;
char *val;
char buf[256];

data = DATA;
if (!data) return;

if (*macro && *(macro+1)) {
/* Longer than 1 char -- use curlies */
snprintf(buf, sizeof(buf), "{%s}", macro);
val = smfi_getsymval(ctx, buf);
} else {
val = smfi_getsymval(ctx, macro);
}
if (!val) return;
dbuf_putc(dbuf, '=');
append_percent_encoded(dbuf, macro);
dbuf_putc(dbuf, ' ');
append_percent_encoded(dbuf, val);
dbuf_putc(dbuf, '\n');
}


at snprintf(buf, sizeof(buf), "{%s}", macro);

that buf[256] is not getting cleaned up even though it is a local
variable that is falling out of scope?

The strike against that theory would be that my memory dump shows that
the "i" and "j" variables are included in that memory chunk.  But the
single character i and j values are never copied to the local buf[256]
variable.

On Fri, Nov 10, 2017 at 12:15 PM, Dianne Skoll  wrote:
> On Fri, 10 Nov 2017 10:33:10 -0500
> Robert Theisen  wrote:
>
> [snip]
>
>> and that macro gets copied to dbuf at line  952
>>
>> /* Write the standard macros */
>> macro = StandardSendmailMacros;
>> while (*macro) {
>> append_macro_value(, ctx, *macro);
>> macro++;
>> }
>>
>>
>> Is it possible that that dbuf is not getting freed properly on down
>> the line?
>
> I can't see how.  A bit further down:
>
> if (data->cmdFD < 0) {
> dbuf_free();
> cleanup(ctx);
> DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
> return SMFIS_TEMPFAIL;
> }
> if (write_dbuf(, data->cmdFD, data, "COMMANDS") < 0) {
> dbuf_free();
> cleanup(ctx);
> DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
> return SMFIS_TEMPFAIL;
> }
> dbuf_free();
>
> and there are no branches or returns between the code you posted
> and the code I posted above.  However, I just checked one of our
> busy servers and the mimedefang process is 8GB which seems large.
> I'll investigate to see if I can find a memory leak.
>
> Regards,
>
> Dianne.
> ___
> NOTE: If there is a disclaimer or other legal boilerplate in the above
> message, it is NULL AND VOID.  You may ignore it.
>
> Visit http://www.mimedefang.org and http://www.roaringpenguin.com
> MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
> http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Dianne Skoll 
On Fri, 10 Nov 2017 10:33:10 -0500
Robert Theisen  wrote:

[snip]

> and that macro gets copied to dbuf at line  952
> 
> /* Write the standard macros */
> macro = StandardSendmailMacros;
> while (*macro) {
> append_macro_value(, ctx, *macro);
> macro++;
> }
> 
> 
> Is it possible that that dbuf is not getting freed properly on down
> the line?

I can't see how.  A bit further down:

if (data->cmdFD < 0) {
dbuf_free();
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
if (write_dbuf(, data->cmdFD, data, "COMMANDS") < 0) {
dbuf_free();
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
dbuf_free();

and there are no branches or returns between the code you posted
and the code I posted above.  However, I just checked one of our
busy servers and the mimedefang process is 8GB which seems large.
I'll investigate to see if I can find a memory leak.

Regards,

Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] Mimedefang with postfix. Process memory usage over time.

2017-11-10 Thread Robert Theisen 
We have an environment with postfix-2.10.1-6.el7.x86_64 and
mimedefang-2.78-6.el7.x86_64 .  Postfix is configured to interact with
mimedefang via smtpd_milters =
unix:/var/spool/MIMEDefang/mimedefang.sock

[root@ext-mx02 ~]# uname -a
Linux ext-mx02 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21
EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@ext-mx02 ~]# free -h
  totalusedfree  shared  buff/cache   available
Mem:15G 10G2.5G692M2.6G4.0G
Swap:  8.1G5.8G2.3G


The mimedefang process looks like this after running for approximately 3 weeks

defang   26810  0.3 42.8 12680732 6961512 ?Sl   Oct17 109:15
/usr/bin/mimedefang -P /var/spool/MIMEDefang/mimedefang.pid -m
/var/spool/MIMEDefang/mimedefang-multiplexor.sock -y -R -1 -U defang
-T -r -H -s -t -G -q -p /var/spool/MIMEDefang/mimedefang.sock


12.1 GB VIRT
6.6 GB RES


I checked the smap data for that mimedefang process and selected one
of the many 64MB blocks that was currently in swap.

7f895c00-7f895fffc000 rw-p  00:00 0
Size:  65520 kB
Rss:   65520 kB
Pss:   65520 kB
Shared_Clean:  0 kB
Shared_Dirty:  0 kB
Private_Clean: 0 kB
Private_Dirty: 65520 kB
Referenced:37632 kB
Anonymous: 65520 kB
AnonHugePages: 0 kB
Swap:  0 kB
KernelPageSize:4 kB
MMUPageSize:   4 kB
Locked:0 kB
VmFlags: rd wr mr mw me nr sd

I did a memory dump of that block and ran strings against it and I got
a quarter of a million lines that look like this ...

_ {auth_authen} {auth_author} {auth_ssf} {auth_type} {cert_issuer}
{cert_subject} {cipher} {cipher_bits} {daemon_name} {daemon_port} i
{if_addr} {if_name} j {mail_addr} {mail_host} {mail_mailer}
{tls_version} {verify} {rcpt_addr} {rcpt_host} {rcpt_mailer}


That seems to correspond to this code snippet in mimedefang.c line 161
(in version 2.78) ...

/* Standard Sendmail macros */
/* We can't make it char const * because libmilter
   is not const-correct. */
static char *StandardSendmailMacros[] = {
"_", "auth_authen", "auth_author", "auth_ssf", "auth_type",
"cert_issuer", "cert_subject", "cipher", "cipher_bits", "daemon_name",
"daemon_port",
"i", "if_addr", "if_name", "j", "mail_addr", "mail_host", "mail_mailer",
"tls_version", "verify", "rcpt_addr", "rcpt_host", "rcpt_mailer",
/* End of macros MUST be marked with NULL! */
NULL
};


and that macro gets copied to dbuf at line  952

/* Write the standard macros */
macro = StandardSendmailMacros;
while (*macro) {
append_macro_value(, ctx, *macro);
macro++;
}


Is it possible that that dbuf is not getting freed properly on down the line?
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang