Re: 4KB of disk space burning on a single email

[Terry Carmen]

Arora, Sumit wrote:

You are right Joost, 4KB will be the block size...
But my question is who is taking this disk space, because email is deleted 
already.
  
Please show Postfix log entries showing the message being delivered to 
the content filter and successfully accepted and deleted as:


Nov 20 13:41:10 wormhole postfix/lmtp[17080]: 81D9430101: 
to=, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, 
delays=0.19/0/0/0.88, dsn=2.0.0

Nov 20 13:41:10 wormhole postfix/qmgr[5151]: 81D9430101: removed

then try

postcat -q 

If it says:

[r...@wormhole ~]# postcat -q 81D9430101
postcat: fatal: open queue file 81D9430101: No such file or directory

then it's not a postfix problem.

Terry

Re: 4KB of disk space burning on a single email

[Ralf Hildebrandt]

* J. Roeleveld :

> The log-entries you sent me are nowhere near 4KB in size.
> How do you check that the "maillog" increases by 4KB per message?

My guess is the mails are marked as deleted. Nothing more. They never
get deleted, thus the space never gets free.

Furthermore it's OT!

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: 4KB of disk space burning on a single email

[J. Roeleveld]

On Friday 20 November 2009 17:01:56 Simon Waters wrote:
> On Friday 20 November 2009 15:52:13 J. Roeleveld wrote:
> > If it's a collection of different logfiles adding up to 4KB in size per
> > email, then you might want to recheck your syslog-configuration.
> 
> Probably pays to know what file system and mount options are applicable
>  here as well.
> 
> As I find it all too believable one could make a filesystem eat a block per
> distinct write these days.
> 
> Now me, I get 24Kb of log file for every message delivered, but that
>  includes messages about rejected spam :(
> 

I also get quite a bit of logfile entries for every message, but that does 
include the filtering/redirecting/sieve/. stuff that's in there as well.

If people want, I can repost the logentries the OP sent me, but they are 
pretty standard as far as I can tell.

--
Joost

Re: 4KB of disk space burning on a single email

[Victor Duchovni]

On Fri, Nov 20, 2009 at 04:01:56PM +, Simon Waters wrote:

> On Friday 20 November 2009 15:52:13 J. Roeleveld wrote:
> > 
> > If it's a collection of different logfiles adding up to 4KB in size per
> > email, then you might want to recheck your syslog-configuration.
> 
> Probably pays to know what file system and mount options are applicable here 
> as well.
> 
> As I find it all too believable one could make a filesystem eat a block per 
> distinct write these days.

If the OP is measuring I/O rates, not disk storage, then the simplest
explanation is failure to turn off synchronous syslog writes.

FAQ (Linux syslog.conf for mail):

mail.info   -/var/log/maillog

the "-" is not optional on systems that handle non-trivial
volumes of email.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:


If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Re: 4KB of disk space burning on a single email

[Simon Waters]

On Friday 20 November 2009 15:52:13 J. Roeleveld wrote:
> 
> If it's a collection of different logfiles adding up to 4KB in size per
> email, then you might want to recheck your syslog-configuration.

Probably pays to know what file system and mount options are applicable here 
as well.

As I find it all too believable one could make a filesystem eat a block per 
distinct write these days.

Now me, I get 24Kb of log file for every message delivered, but that includes 
messages about rejected spam :(

Re: 4KB of disk space burning on a single email

[J. Roeleveld]

On Friday 20 November 2009 14:53:58 Arora, Sumit wrote:
> I've checked the size of maillog
> It's size is getting increased by 4KB on receiving each email.
> 
> Noel Jones, please read the other emails of this thread.
> 
> -Sumit Arora

The log-entries you sent me are nowhere near 4KB in size.
How do you check that the "maillog" increases by 4KB per message?

Did you use the suggested method to check where the increases all occur?

It might easily be a collection of different logfiles and/or temp-files that 
are being kept.
If it's a collection of different logfiles adding up to 4KB in size per email, 
then you might want to recheck your syslog-configuration.

--
Joost

Re: 4KB of disk space burning on a single email

[Noel Jones]

On 11/20/2009 7:53 AM, Arora, Sumit wrote:

I've checked the size of maillog
It's size is getting increased by 4KB on receiving each email.

Noel Jones, please read the other emails of this thread.

-Sumit Arora


I did.  All I see are unsupported top-posted claims.

  -- Noel Jones

Re: 4KB of disk space burning on a single email

[Wietse Venema]

Arora, Sumit:
> I've checked the size of maillog
> It's size is getting increased by 4KB on receiving each email.

We know what you believe.

Now we would like to see some actual concrete evidence.

Wietse

RE: 4KB of disk space burning on a single email

[Arora, Sumit]

I've checked the size of maillog
It's size is getting increased by 4KB on receiving each email.

Noel Jones, please read the other emails of this thread.

-Sumit Arora

-Original Message-
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of Noel Jones
Sent: Friday, November 20, 2009 7:20 PM
To: postfix-users@postfix.org
Subject: Re: 4KB of disk space burning on a single email

On 11/20/2009 7:44 AM, Arora, Sumit wrote:
> It's being passed for content filtering and email is not sent back to postfix.
> For postfix it's just mail forwarding
>

Amavisd-new (and probably other content filters too) keeps 
temporary files between messages to prevent creating new files 
every time a mail comes in.

Sounds as if you're chasing a problem that doesn't exist.

  -- Noel Jones

Re: 4KB of disk space burning on a single email

[Noel Jones]

On 11/20/2009 7:44 AM, Arora, Sumit wrote:

It's being passed for content filtering and email is not sent back to postfix.
For postfix it's just mail forwarding



Amavisd-new (and probably other content filters too) keeps 
temporary files between messages to prevent creating new files 
every time a mail comes in.


Sounds as if you're chasing a problem that doesn't exist.

 -- Noel Jones

RE: 4KB of disk space burning on a single email

[Arora, Sumit]

It's being passed for content filtering and email is not sent back to postfix.
For postfix it's just mail forwarding

-Sumit Arora

-Original Message-
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of Ralf Hildebrandt
Sent: Friday, November 20, 2009 7:11 PM
To: postfix-users@postfix.org
Subject: Re: 4KB of disk space burning on a single email

* Arora, Sumit :
> You are right Joost, 4KB will be the block size...
> But my question is who is taking this disk space, because email is deleted 
> already.

How EXACTLY is the email being deleted ?

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: 4KB of disk space burning on a single email

[J. Roeleveld]

On Friday 20 November 2009 14:38:50 Arora, Sumit wrote:
> I'm really amazed... its maillog that is consuming 4KB on a simple 5-7 line
>  logging for each email
> 
> -Sumit Arora

How long are those lines?

> 
> -Original Message-
> From: owner-postfix-us...@postfix.org
>  [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent:
>  Friday, November 20, 2009 6:51 PM
> To: postfix-users@postfix.org
> Subject: Re: 4KB of disk space burning on a single email
> 
> On Friday 20 November 2009 14:14:03 Arora, Sumit wrote:
> > You are right Joost, 4KB will be the block size...
> > But my question is who is taking this disk space, because email is
> > deleted already.
> >
> > Is this any log or what I just have to stop consumption of any kind
> > of disk space except mail logs.
> >
> > Please help
> >
> > -Sumit Arora
> 
> 4K on logs per email are quite a lot. Unless you have really verbose
>  logging. Unless your system logger is duplicating messages over several
>  logfiles?
> 
> Possible places to look:
> /var/spool/.
> /tmp/
> /var/tmp/
> 
> Maybe something leave temporary files, or maybe your postfix is keeping
>  files in a spool-directory.
> 
> Maybe try the following to see where additional files are placed:
> 
> du / > PRE-newemail.txt
> 
> du / > POST-newemail.txt
> 
> Then do a diff on these 2 text-files to see which directories had an
>  increase. Hopefully, this will reduce the possible locations to check for
>  possible locations.
> 
> --
> Joost
> 
> > -Original Message-
> > From: owner-postfix-us...@postfix.org
> >  [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent:
> >  Friday, November 20, 2009 6:26 PM
> > To: postfix-users@postfix.org
> > Subject: Re: 4KB of disk space burning on a single email
> >
> > On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> > > Hi folks,
> > >
> > > I'm just testing my postfix server for load and disk usage.
> > > I'm using content filter on some another server, and I don't feed the
> > > email back to postfix.
> > >
> > > Email data is getting deleted from my postfix server, but I don't know
> > >  where 4KB of my disk space gone on every email my postfix server
> > > receives. If anybody have some idea, please tell me.
> > >
> > > -Sumit Arora
> >
> > My guess is that the filesystem has a block-size of 4KB.
> >
> > A file is stored over a set of blocks of this size. Which means that if a
> >  file is smaller then 4KB, it will still use a 4KB block.
> >
> > As far as I know, only reiserfs has the "notail"option which will try to
> >  stick multiple smaller files into a single block.
> >
> > --
> > Joost
> 

Re: 4KB of disk space burning on a single email

[Ralf Hildebrandt]

* Arora, Sumit :
> You are right Joost, 4KB will be the block size...
> But my question is who is taking this disk space, because email is deleted 
> already.

How EXACTLY is the email being deleted ?

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

RE: 4KB of disk space burning on a single email

[Arora, Sumit]

I'm really amazed... its maillog that is consuming 4KB on a simple 5-7 line 
logging for each email

-Sumit Arora

-Original Message-
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of J. Roeleveld
Sent: Friday, November 20, 2009 6:51 PM
To: postfix-users@postfix.org
Subject: Re: 4KB of disk space burning on a single email

On Friday 20 November 2009 14:14:03 Arora, Sumit wrote:
> You are right Joost, 4KB will be the block size...
> But my question is who is taking this disk space, because email is deleted
>  already.
> 
> Is this any log or what I just have to stop consumption of any kind of
>  disk space except mail logs.
> 
> Please help
> 
> -Sumit Arora

4K on logs per email are quite a lot. Unless you have really verbose logging.
Unless your system logger is duplicating messages over several logfiles?

Possible places to look:
/var/spool/.
/tmp/
/var/tmp/

Maybe something leave temporary files, or maybe your postfix is keeping files 
in a spool-directory.

Maybe try the following to see where additional files are placed:

du / > PRE-newemail.txt

du / > POST-newemail.txt

Then do a diff on these 2 text-files to see which directories had an increase.
Hopefully, this will reduce the possible locations to check for possible 
locations.

--
Joost

> 
> -Original Message-
> From: owner-postfix-us...@postfix.org
>  [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent:
>  Friday, November 20, 2009 6:26 PM
> To: postfix-users@postfix.org
> Subject: Re: 4KB of disk space burning on a single email
> 
> On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> > Hi folks,
> >
> > I'm just testing my postfix server for load and disk usage.
> > I'm using content filter on some another server, and I don't feed the
> > email back to postfix.
> >
> > Email data is getting deleted from my postfix server, but I don't know
> >  where 4KB of my disk space gone on every email my postfix server
> > receives. If anybody have some idea, please tell me.
> >
> > -Sumit Arora
> 
> My guess is that the filesystem has a block-size of 4KB.
> 
> A file is stored over a set of blocks of this size. Which means that if a
>  file is smaller then 4KB, it will still use a 4KB block.
> 
> As far as I know, only reiserfs has the "notail"option which will try to
>  stick multiple smaller files into a single block.
> 
> --
> Joost
> 

Re: 4KB of disk space burning on a single email

[J. Roeleveld]

On Friday 20 November 2009 14:14:03 Arora, Sumit wrote:
> You are right Joost, 4KB will be the block size...
> But my question is who is taking this disk space, because email is deleted
>  already.
> 
> Is this any log or what I just have to stop consumption of any kind of
>  disk space except mail logs.
> 
> Please help
> 
> -Sumit Arora

4K on logs per email are quite a lot. Unless you have really verbose logging.
Unless your system logger is duplicating messages over several logfiles?

Possible places to look:
/var/spool/.
/tmp/
/var/tmp/

Maybe something leave temporary files, or maybe your postfix is keeping files 
in a spool-directory.

Maybe try the following to see where additional files are placed:

du / > PRE-newemail.txt

du / > POST-newemail.txt

Then do a diff on these 2 text-files to see which directories had an increase.
Hopefully, this will reduce the possible locations to check for possible 
locations.

--
Joost

> 
> -Original Message-
> From: owner-postfix-us...@postfix.org
>  [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent:
>  Friday, November 20, 2009 6:26 PM
> To: postfix-users@postfix.org
> Subject: Re: 4KB of disk space burning on a single email
> 
> On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> > Hi folks,
> >
> > I'm just testing my postfix server for load and disk usage.
> > I'm using content filter on some another server, and I don't feed the
> > email back to postfix.
> >
> > Email data is getting deleted from my postfix server, but I don't know
> >  where 4KB of my disk space gone on every email my postfix server
> > receives. If anybody have some idea, please tell me.
> >
> > -Sumit Arora
> 
> My guess is that the filesystem has a block-size of 4KB.
> 
> A file is stored over a set of blocks of this size. Which means that if a
>  file is smaller then 4KB, it will still use a 4KB block.
> 
> As far as I know, only reiserfs has the "notail"option which will try to
>  stick multiple smaller files into a single block.
> 
> --
> Joost
> 

Re: 4KB of disk space burning on a single email

[Dan Schaefer]

Arora, Sumit wrote:

You are right Joost, 4KB will be the block size...
But my question is who is taking this disk space, because email is deleted 
already.

Is this any log or what I just have to stop consumption of any kind of disk 
space except mail logs.

Please help

-Sumit Arora

-Original Message-
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of J. Roeleveld
Sent: Friday, November 20, 2009 6:26 PM
To: postfix-users@postfix.org
Subject: Re: 4KB of disk space burning on a single email

On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
  

Hi folks,

I'm just testing my postfix server for load and disk usage.
I'm using content filter on some another server, and I don't feed the email
 back to postfix.

Email data is getting deleted from my postfix server, but I don't know
 where 4KB of my disk space gone on every email my postfix server receives.
 If anybody have some idea, please tell me.

-Sumit Arora




My guess is that the filesystem has a block-size of 4KB.

A file is stored over a set of blocks of this size. Which means that if a file 
is smaller then 4KB, it will still use a 4KB block.


As far as I know, only reiserfs has the "notail"option which will try to stick 
multiple smaller files into a single block.


--
Joost
  
Perhaps tell your mail client to "compact" the trash. That's what I have 
to do in Thunderbird, or else the files stay on the server.


Dan Schaefer
Web Developer/Systems Analyst
Performance Administration Corp.

RE: 4KB of disk space burning on a single email

[Arora, Sumit]

You are right Joost, 4KB will be the block size...
But my question is who is taking this disk space, because email is deleted 
already.

Is this any log or what I just have to stop consumption of any kind of disk 
space except mail logs.

Please help

-Sumit Arora

-Original Message-
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of J. Roeleveld
Sent: Friday, November 20, 2009 6:26 PM
To: postfix-users@postfix.org
Subject: Re: 4KB of disk space burning on a single email

On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> Hi folks,
> 
> I'm just testing my postfix server for load and disk usage.
> I'm using content filter on some another server, and I don't feed the email
>  back to postfix.
> 
> Email data is getting deleted from my postfix server, but I don't know
>  where 4KB of my disk space gone on every email my postfix server receives.
>  If anybody have some idea, please tell me.
> 
> -Sumit Arora
> 

My guess is that the filesystem has a block-size of 4KB.

A file is stored over a set of blocks of this size. Which means that if a file 
is smaller then 4KB, it will still use a 4KB block.

As far as I know, only reiserfs has the "notail"option which will try to stick 
multiple smaller files into a single block.

--
Joost

Re: 4KB of disk space burning on a single email

[Wietse Venema]

J. Roeleveld:
> On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> > Hi folks,
> > 
> > I'm just testing my postfix server for load and disk usage.
> > I'm using content filter on some another server, and I don't feed the email
> >  back to postfix.
> > 
> > Email data is getting deleted from my postfix server, but I don't know
> >  where 4KB of my disk space gone on every email my postfix server receives.
> >  If anybody have some idea, please tell me.
> > 
> > -Sumit Arora
> > 
> 
> My guess is that the filesystem has a block-size of 4KB.
> 
> A file is stored over a set of blocks of this size. Which means
> that if a file is smaller then 4KB, it will still use a 4KB block.
> 
> As far as I know, only reiserfs has the "notail"option which will
> try to stick multiple smaller files into a single block.

25 Years ago, the UFS file system came with large blocks (typ 8k)
and small fragments (typ 1k).  The "tail fragments" from different
files could then share the same large block.  In the mean time,
the 8k blocksize has been replaced by larger blocksizes like 32k
but the idea is the same: low fragmentation without wasting space.

Wietse

Marshall Kirk McKusick, William N. Joy, Samuel J. Leffler and Robert
S. Fabry. A Fast File System for UNIX. Technical Report Computer
Systems Research Group, Computer Science Division, Department of
Electrical Engineering and Computer Science, University of California,
Berkeley, Berkeley, CA 94720.
http://www.cs.berkeley.edu/~brewer/cs262/FFS.pdf

Re: 4KB of disk space burning on a single email

[J. Roeleveld]

On Friday 20 November 2009 13:42:58 Arora, Sumit wrote:
> Hi folks,
> 
> I'm just testing my postfix server for load and disk usage.
> I'm using content filter on some another server, and I don't feed the email
>  back to postfix.
> 
> Email data is getting deleted from my postfix server, but I don't know
>  where 4KB of my disk space gone on every email my postfix server receives.
>  If anybody have some idea, please tell me.
> 
> -Sumit Arora
> 

My guess is that the filesystem has a block-size of 4KB.

A file is stored over a set of blocks of this size. Which means that if a file 
is smaller then 4KB, it will still use a 4KB block.

As far as I know, only reiserfs has the "notail"option which will try to stick 
multiple smaller files into a single block.

--
Joost