Re: [Dovecot] Temporary files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 14 Jul 2010, Timo Sirainen wrote: But /var/tmp exists, right? So, just in case, the default could be: /var/tmp, but /tmp if /var/tmp is missing. Personally, I just would use /tmp, because these file do not need to survive a machine reboot, right? Regards, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBTEAV+7+Vh58GPL/cAQLovQf9HfHw7iSooDPgxq8Q9UK9MEbr8tbu7q8u WeCAQQ+s1PH7sVmcUmrdn9OdYB7DEse3tcGwgkZ7da5z7kfnWySC72u4JeoRmJZl fCYOtA7AcE5Fll42G3vaLbMXOg3pymcpb2XMT3aDG5ABeNkTFJXzn7ydifaki7+Y q9wpf/Mk+I545yvplorZOTRVZHxfXbmtoP7YJO21z+pmzQUuAYPkJ8psOk556cMN l81UGA2X8nQDL8ZDPdkhmF0t1nddzppjF41WS20tCTVeAi87bS2wMC1mM9Fgxyl4 A+Z8t+r40Ey+RjGpY0WfUInQt0QqaxHcQxqsnqvkUu/qwh9fOvR8wg== =bRuO -END PGP SIGNATURE-
Re: [Dovecot] Temporary files
Hey all, On 07/14/2010 11:02 PM, Timo Sirainen wrote: On Wed, 2010-07-14 at 22:55 +0200, Edgar Fuß wrote: Any thoughts? Could the location be made dependant on the file size? The file size isn't known beforehand. But it could be known if the SIZE extention of LMTP is used right? Talking about LMTP ... if one runs lets say postfix on the deliver machine that is already maintaining a mailqueue and one uses the LMTP server of dovecot 2.0 to get the messages delivered, why would you need to write the mail to disk a second time (postfix once, then deliver again) of the message size is within reasonable limits (lets say below 100MB). So would you make the size when messages are spooled to disk (currently 128k as I understand) configurable? Regards, Christian
Re: [Dovecot] Temporary files
On Wed, Jul 14, 2010 at 11:53:48PM -0500, Stan Hoeppner wrote: Don't rely on an existing temp directory. Create /var/spool/dovecot/ and /var/spool/dovecot/tmp/ with the installation script and assign the appropriate permissions. This solves the current problem and gives you a dedicated dovecot spool directory to use in the future for other functions/features. +1
Re: [Dovecot] Temporary files
1.
If know you are delivering to Maildir, and you know the folder you are
delivering to, then surely you have a temporary directory in the tmp
out of Maildir's {new,cur,tmp} ? And also the potential benefit of being
able to rename the resulting file into the new directory.
2.
I would like to have the temporary directory be configurable (but with a
sane default like /tmp or perhaps /tmp/dovecot).
In an installation where Dovecot is being used heavily, people might
want to assign specific resources for Dovecot operation. So someone
might decide that Dovecot should have a specific device (hard disk /
solid state disk / whatever) for its temporary files, other than the
default /tmp.
Bill
Re: [Dovecot] Temporary files
On Wed, Jul 14, 2010 at 09:34:52PM +0100, Timo Sirainen wrote: [...] So why was the move made in the first place? Because a) some people had small (maybe ramfs) /tmp and Dovecot was eating it all up and b) to On Solaris, this is the situation, though it's not called ramfs but: l...@hydra:/tmp$ uname -a SunOS hydra 5.10 Generic_141444-09 sun4v sparc SUNW,Sun-Fire-T200 l...@hydra:/tmp$ df -h . Filesystem size used avail capacity Mounted on swap16G 0K16G 0%/tmp l...@hydra:/tmp$ swap -l swapfile dev swaplo blocks free /dev/zvol/dsk/rpool/swap 256,1 16 33554416 33554416 Yes, this machine has some RAM and enough swap zvol as well, but still there can be machines with less resources too, and in Solaris /tmp is not a normal filesystem anyway ... /var/tmp is available though, and it's a normal filesystem ...
Re: [Dovecot] Temporary files
On 15.7.2010, at 9.16, Christian Rohmann wrote: Any thoughts? Could the location be made dependant on the file size? The file size isn't known beforehand. But it could be known if the SIZE extention of LMTP is used right? It wouldn't be known with deliver. And I wasn't planning on implementing SIZE extension for LMTP anytime soon.. Talking about LMTP ... if one runs lets say postfix on the deliver machine that is already maintaining a mailqueue and one uses the LMTP server of dovecot 2.0 to get the messages delivered, why would you need to write the mail to disk a second time (postfix once, then deliver again) of the message size is within reasonable limits (lets say below 100MB). So would you make the size when messages are spooled to disk (currently 128k as I understand) configurable? Temporary files would typically stay in memory too and get deleted before written to disk. But I guess I could make this configurable.
Re: [Dovecot] Temporary files
On 15.7.2010, at 10.02, William Blunn wrote:
If know you are delivering to Maildir, and you know the folder you are
delivering to,
But I don't. Sieve could do just about anything to the message.
then surely you have a temporary directory in the tmp out of Maildir's
{new,cur,tmp} ? And also the potential benefit of being able to rename the
resulting file into the new directory.
This is a very Maildir-specific optimization that doesn't work with anything
else.
I would like to have the temporary directory be configurable (but with a sane
default like /tmp or perhaps /tmp/dovecot).
Sure.
Re: [Dovecot] Temporary files
The file size isn't known beforehand. I thought that because it's known to be 128k, it's known.
Re: [Dovecot] Temporary files
On Thu, 2010-07-15 at 14:52 +0200, Edgar Fuß wrote: The file size isn't known beforehand. I thought that because it's known to be 128k, it's known. Dovecot reads 128 kB to memory first, then thinks this is beginning to be too much and starts writing to disk instead.
Re: [Dovecot] Temporary files
On 7/15/2010 1:27 AM, Thomas Leuxner wrote: On Wed, Jul 14, 2010 at 11:53:48PM -0500, Stan Hoeppner wrote: Don't rely on an existing temp directory. Create /var/spool/dovecot/ and /var/spool/dovecot/tmp/ with the installation script and assign the appropriate permissions. This solves the current problem and gives you a dedicated dovecot spool directory to use in the future for other functions/features. +1 +2. Right from the FHS section - /var/spool contains data which is awaiting some kind of later processing. Data in /var/spool represents work to be done in the future (by a program, user, or administrator); often data is deleted after it has been processed. Having a configuration parameter as well is always good - but make /var/spool/dovecot the default. -- Daniel
Re: [Dovecot] Temporary files
On Thu, 2010-07-15 at 08:54 -0700, Daniel L. Miller wrote: /var/spool contains data which is awaiting some kind of later processing. Data in /var/spool represents work to be done in the future (by a program, user, or administrator); often data is deleted after it has been processed. I don't think this is what the files are. They are created and they're immediately unlinked before anything is even written to them. The files are treated kind of like being virtual memory, but since they're stored in filesystem rather than kept in memory, the kernel knows that under memory pressure it's better to write them to disk than move memory pages to swap. So there is no awaiting or work to be done in future. And data isn't deleted after it has been processed, it's deleted even before processing begins.. BTW. I've already committed the mail_temp_dir with a default to /tmp. I forgot also that nowadays if writing to the temp file fails, the mail delivery won't fail but instead everything is moved into memory. So a default /tmp isn't that bad anymore.
Re: [Dovecot] Temporary files
Any thoughts? Could the location be made dependant on the file size? My usual setup is mfs on /tmp and ffs on /var/tmp, so /tmp being smaller and faster and /var/tmp being larger and slower.
Re: [Dovecot] Temporary files
On 7/14/10 4:34 PM, Timo Sirainen wrote: In v1.0 .. v1.1 deliver was writing incoming 128k mail to /tmp file (to avoid reading it all into memory). In v1.2 I moved it to user's home directory. This slowed deliveries for NFS users. Also people with filesystem quota had trouble since now user required twice as much available quota to save a message. The FS quota problem was solved by having quota-fs plugin change the default temp file path to /tmp. So why was the move made in the first place? Because a) some people had small (maybe ramfs) /tmp and Dovecot was eating it all up and b) to avoid any potential security issues with storing per-user files to a shared directory. I haven't seen anywhere that /tmp is actually a ramfs, but that is besides your point. I would say that if people are using Dovecot they should be designing the partitions around the need for the software and services they are running (at least that is what I do). I'm now thinking about a new setting (mail_temp_dir?) for v2.0 where these temp files would be stored (lda, lmtp, some others in future). Maybe the b) problem isn't that huge, so maybe there doesn't need to be a way to store them to user's home. So that leaves a) problem. Maybe the setting could default to /var/tmp instead of /tmp? That probably has more disk space available usually. (I hope /var/tmp is available everywhere?) Everywhere I have seen /var/tmp used, it has been linked to /tmp. I would recommend using /tmp as the default. Also, not all distributions, including Slackware (which a lot of slack users like Dovecot) do use /var/tmp for anything, which can get confusing. I really like the idea of a default, and then the ability to change the default. That way if people need to grow out of /tmp or wherever you decide the default is going to be, it can be moved easily. Any thoughts? -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: [Dovecot] Temporary files
On Wed, 2010-07-14 at 22:55 +0200, Edgar Fuß wrote: Any thoughts? Could the location be made dependant on the file size? The file size isn't known beforehand. It's between 128 kB and whatever the max. mail size is configured to be in your system.
Re: [Dovecot] Temporary files
Den 14.07.2010 22:34, skrev Timo Sirainen: In v1.0 .. v1.1 deliver was writing incoming128k mail to /tmp file (to avoid reading it all into memory). In v1.2 I moved it to user's home directory. This slowed deliveries for NFS users. Also people with filesystem quota had trouble since now user required twice as much available quota to save a message. The FS quota problem was solved by having quota-fs plugin change the default temp file path to /tmp. So why was the move made in the first place? Because a) some people had small (maybe ramfs) /tmp and Dovecot was eating it all up and b) to avoid any potential security issues with storing per-user files to a shared directory. I'm now thinking about a new setting (mail_temp_dir?) for v2.0 where these temp files would be stored (lda, lmtp, some others in future). Maybe the b) problem isn't that huge, so maybe there doesn't need to be a way to store them to user's home. So that leaves a) problem. Maybe the setting could default to /var/tmp instead of /tmp? That probably has more disk space available usually. (I hope /var/tmp is available everywhere?) Any thoughts? I'd like to have /tmp as a default, but a variable to set it if the need is there. Much like default /tmp on MySQL works a long time, until you need to ALTER TABLE on that 30 GB table ;) Arne
Re: [Dovecot] Temporary files
On 14.7.2010, at 22.02, Steven King wrote: So why was the move made in the first place? Because a) some people had small (maybe ramfs) /tmp and Dovecot was eating it all up and b) to avoid any potential security issues with storing per-user files to a shared directory. I haven't seen anywhere that /tmp is actually a ramfs, but that is besides your point. I would say that if people are using Dovecot they should be designing the partitions around the need for the software and services they are running (at least that is what I do). But there are tons of people who just use defaults, at least until they run into problems, so the default needs to be a good one. In this case I'd even say probably over 99% of Dovecot users won't change the default, and most of them not even aware of the existence of this setting.. Everywhere I have seen /var/tmp used, it has been linked to /tmp. I don't see that in any systems I have access to right now.. FHS says /var/tmp is for temporary files that must not be removed when rebooting a machine. So if /tmp is a ramfs, /var/tmp is almost guaranteed not to be. I would recommend using /tmp as the default. Also, not all distributions, including Slackware (which a lot of slack users like Dovecot) do use /var/tmp for anything, which can get confusing. But /var/tmp exists, right?
Re: [Dovecot] Temporary files
Timo Sirainen put forth on 7/14/2010 3:34 PM: Any thoughts? Don't rely on an existing temp directory. Create /var/spool/dovecot/ and /var/spool/dovecot/tmp/ with the installation script and assign the appropriate permissions. This solves the current problem and gives you a dedicated dovecot spool directory to use in the future for other functions/features. -- Stan
