Re: Moving /tmp to tmpfs is fine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Friday 25 May 2012 05:55 PM, Serge wrote: > So instead of fixing the defaults you suggest everybody to drop > the programs they use (mc, firefox, mysql)? ;) I think I'll agree with you here. The current state seems to be broken. Having tmpfs on /tmp is fine but then we need to fix the values for os.tmpfile(), TMPDIR et cetera. Hopefully targeting it for Wheezy. Regarding the big file for /var/tmp and small file for /tmp, I think that does not make much sense, unless you have os.bigtmpfile() and os.smltmpfile(). If I (human) know it is "tmp", I would rather throw it in Trash. - -- Given the large number of mailing lists I follow, I request you to CC me in replies for quicker response -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCgAGBQJPy7zxAAoJEKY6WKPy4XVpdKUP/3ckvjVlSQF0sMXLSKZkhhbq m2+arwSMJyfCRGeVgajymjAlXtUhsqJJ1FKaqY1eePl4G/PZAI36zlxImzZzxHSn jq7hgPfjoQ4QMlNMXCIJAHBnLX1GjoQi0MaA0F3ZoHJQ0dkofThfWwwKZbGQHL4g kjk3xPzmHVPBTGZ08bpZB2j6BIYvWdmiR1QcQnLjMkrmaTpStB+VRPHDFJIX5FFT RlYQ97xcu951pqVF2j2fUtyZ/0eh+2RQ3+EkBilmBCgYLBZKnBZgcNiRja0CWIIp 35y6iGB1h1HopS75qc8ymQyIZ6WmVlKZFDot9wtdB8IxYHYNNkHWPmNBeQF5xoOL rNS4BJE9DjBB/FdA51fs4v6G9mkmrkOiuk6ZzyuDyCEd1vwNQgccQqTNQrjwPTY0 3oIPPa/GQ3fHWbQS3Nq7y/elD19x0pGxe0FQCQLCfs1SNm7qyzPaj2G4JIZ5cTsy JCq2zBLsm9Ds350G1DVSg6onY/u4C/D2sQC64iZehp4aCpbCYOPN19gUiD0/o+f1 JrZhfyGA37muvBKcpWqgdUgNqHtKkz8C1T8nIOJg1+GsJCAvETZYiWDkqOmDF1rX 13wHSIrFMcW/9gMZUcGWa6jTBxofEPqdAnIOGU20Prn94MJnfz/8kpARDcAF6rQ/ 3mQMG0O6ASr2D8wJRURY =Gu99 -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fcbbcf2.4030...@researchut.com
Re: Moving /tmp to tmpfs is fine
2012/5/30 Thomas Goirand wrote: >> You mean you know some real applications becoming noticeably faster >> having /tmp on tmpfs? > > No, I mean that writing "nobody can notice that on real applications" > is a bit too extreme, that's all I'm saying. Right, sorry. Of course, I may be wrong, there can be such applications. It's just nobody had suggested them yet. -- Just trying to make the world better, Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caovenepexlsxzgvqabqqd-1mr-exryqh4c1kddhno3ppfxw...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
On 05/30/2012 10:07 AM, Serge wrote: > 2012/5/28 Thomas Goirand wrote: > > >>> The truth is that tmpfs IS FASTER in some cases. The problem is that >>> *nobody* can notice that on *real* applications. >>> >> Serge, I'm on your side of the discussion, but the above is simply >> not truth. >> > You mean you know some real applications becoming noticeably faster > having /tmp on tmpfs? > No, I mean that writing "nobody can notice that on real applications" is a bit too extreme, that's all I'm saying. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc5905b.3090...@debian.org
Re: Moving /tmp to tmpfs is fine
2012/5/27 Ben Hutchings wrote: >> then /tmp using tmpfs *will* lead to issues that many wont understand. > As will /tmp on a small root partition. > As will a small dedicated /tmp partition. True. But debian does not have small root partition *by default*. And it does not install with a small dedicated /tmp partition *by default* as well. Then why does it installs with /tmp on tmpfs *by default*? Users don't like thinking about partitions size, and they don't care much about fragmentation or number of I/O. The most common configuration would probably be entire system on a single partition, optionally in dual-boot with other OSes (i.e. there could be other FAT/NTFS partitions there). TMPDIR on a root partition is the best and only option there. It's good for large files, isn't limited by memory size, don't cause system swapping and is as fast as cached local storage can be. > A shared /tmp also results in various security problems Agree, having a shared /tmp may sometimes be an interesting idea, but it is also bad to have it *by default* > We should be thinking about implementing per-user temporary directories There're a lot of other places where TMPDIR can be. Let's consider all of them. Where else we can put tmp directory: 1. /tmp (TMPDIR on a root partition) Good for systems of regular users. But bad for systems with read-only or small root partition (rare, *never* happens in default install, but still happens). 2. /home/tmp (TMPDIR on a home partition) Good for systems that want users to obey /home quotas or having read-only root. But bad for shared (NFS) /home (not that rare, but still possible). Also bad for users with separate /home partition, who care about their data: when / dies you can recover data from your /home, and tmp on /home will just increase chances of /home death. 3. /home/USER/tmp Same as above, but in addition it's also bad for ssh servers where users share files over /tmp directory (they just don't have a choice there). 4. /bin/tmp Not really useful. But if we name it /bin/trash... hey, we have a trash in a bin! ;) 5. /var/tmp Oh, we already have that. 6. /tmp on a separate partition Common for servers, eats no RAM, have all the limits/quotas working. But needs some brain work and complex in resizing, so it's bad for regular users. 7. /OTHERDIR/tmp Same as above, if OTHERDIR on a separate partition. And same as #1 if OTHERDIR is on a root partition. 8. /tmp on tmpfs Bad for large files. Bad for low-memory systems (i.e. routers, cheap netbooks, tablet PCs, mobiles), which are more and more common nowadays. But still the only option for systems that have slow (network-mounted) /home and /var partitions, and read-only root, and need to work FAST with a lot of small files (I've never seen such case myself, but it is still possible anyway). As you can see each of them have problems. So the one, that causes the fewest problems in default setup should be the default. Isn't it? People, with some unusual installations are customizing their defaults anyway, so they can change /tmp location themselves as well (i.e. do a `mount --bind /tmp /home/tmp` for read-only setup). > We should be thinking about implementing per-user temporary directories > and making sure that programs respect $TMPDIR. That's not enough. Applications need a special temporary directory. The one that is cleaned on reboot. Of course apps should clean their files themselves, but they should not worry about cleaning files after i.e. an unexpected power outage. That's what /tmp is good for. So if you click on a few files in firefox and then your lights turn off, on next reboot firefox should not look through $TMPDIR and delete files that were created by it on last boot. Of course we can write an additional init-script that will automatically clean all user-specific TMPDIRs, it would have to be a very smart script, it should work for local /home as well as i.e. NFS /home, and must not clean all the directories when one of machines reboots. It should probably support things like LDAP, or unusual home-directories like /var/guest. It will surely require a lot of work and probably will cause a lot of problems. But it is possible... I don't see what for, but it's possible. Or maybe we can just leave /tmp as a part of / partition *by default*, since by default it's large enough, cleaned on boot, works for all programs and causes no worries? ;) > I'm not sure whether that's compatible with historical use of /tmp by > the X window system.) I guess it's not. Because when X starts you don't know what user will log in there. PS: see also idea about on-demand mount of /tmp to tmpfs for systems with disk space check. It would make the defaults working even for small and read-only root fs. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caovenerwrhneye0tbktzr58myipege+75xqrf
Re: Moving /tmp to tmpfs is fine
2012/5/28 Thomas Goirand wrote: >> The truth is that tmpfs IS FASTER in some cases. The problem is that >> *nobody* can notice that on *real* applications. > > Serge, I'm on your side of the discussion, but the above is simply > not truth. You mean you know some real applications becoming noticeably faster having /tmp on tmpfs? > And by the way, that's not the issue. The issue is potential > breakage, which we want to avoid *at all costs*. That does not work. I tried that. When I say "Hey, a lot of software breaks because of your change", I get the answer "It's not my change in fault, it's the software, go fix it". This is exactly what I got in that thread, by the way. But when I say "Hey, your change have broken a lot of software and brought nothing good", then people start thinking, why should they mess with fixing a lot of software, if they get nothing in return anyway. This is what we have with "/tmp on tmpfs" change. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOVenErJhn54esbuqxn6HHARXPJMeDj4WzxCuSf3KXm=n=_...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
Miles Bader writes: > bazillion packages in debian that blithely cache vast quantities of > (often very uninteresting) data in random subdirs of $HOME... and then Fortunately there is some movement towards the use of XDG_CACHE_DIR (defaults to ~/.cache). -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/84wr3vk0id@sauna.l.org
Re: Moving /tmp to tmpfs is fine
Thomas Goirand writes: >> Perhaps it should be >> extended to allow the directory to be below ~/ instead of below >> /tmp/. :) >> > I don't think so. As other pointed out, your /home could be > remote (over NFS?), and then slow, while /tmp is normally > on your local machine, so moving the temp folder in ~/ will > potentially slow them down, and break quota. ... yeah, my homedir is in NFS, and I get no end of grief from the bazillion packages in debian that blithely cache vast quantities of (often very uninteresting) data in random subdirs of $HOME... and then expect accessing it to be very fast :[ -miles -- Year, n. A period of three hundred and sixty-five disappointments. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87d35na73z@catnip.gol.com
Re: Moving /tmp to tmpfs is fine
On Tue, 2012-05-29 at 12:15 +0800, Thomas Goirand wrote: > What's the folder structure in /tmp then? /tmp//$USER? It's the Wild West over there. You'll often see something like /tmp/$procname/$pid/blah or /tmp/$procname/$user/blah, or just /tmp/$some_hash_of_who_knows_what/blah. FHS is uncharacteristically laconic: http://www.pathname.com/fhs/pub/fhs-2.3.html#TMPTEMPORARYFILES Weldon -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1338267368.2617.0.camel@minerva
Re: Moving /tmp to tmpfs is fine
]] Thomas Goirand > What's the folder structure in /tmp then? /tmp//$USER? /tmp/user/$UID is the default. It can be overridden, but not in a manner that's compatible with Petter's suggestion. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87mx4rvblx@xoog.err.no
Bug#674984: Moving /tmp to tmpfs is fine
Package: libpam-tmpdir Severity: wishlist ]] Petter Reinholdtsen > [Ben Hutchings] > > We should be thinking about implementing per-user temporary directories > > and making sure that programs respect $TMPDIR. > > Yes, per-user temp directories is a good idea. Installing the > libpam-tmpdir package enable this by default, and beside some problems > with the root user (bad TMPDIR is inherited when I restart services > using /etc/init.d/ scripts), it work well. Perhaps it should be > extended to allow the directory to be below ~/ instead of below > /tmp/. :) Thanks for this suggestion, filing it in the BTS. (JFTR, it won't be the default, but I don't see the harm in making it an option.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87r4u3vbpb@xoog.err.no
Re: Moving /tmp to tmpfs is fine
On 05/29/2012 03:13 AM, Petter Reinholdtsen wrote: > Perhaps it should be > extended to allow the directory to be below ~/ instead of below > /tmp/. :) > I don't think so. As other pointed out, your /home could be remote (over NFS?), and then slow, while /tmp is normally on your local machine, so moving the temp folder in ~/ will potentially slow them down, and break quota. What's the folder structure in /tmp then? /tmp//$USER? Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc44d4e.6030...@debian.org
Re: Moving /tmp to tmpfs is fine
On 05/28/2012 04:47 PM, Bernhard R. Link wrote: > I personally think having tmpdir on /tmp might be a good default for > new systems. If systems get changed to that from something else on > upgrade without asking, I consider that quite an ugly bug. > And you're not the only one. It seems that at least one member of the release team (IMO rightly) think this way as well. See #674517. BTW, I don't think it was possible to hide the RAMTMP variable/switch better than it is right now in SID... :) Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc3d1ff.8020...@debian.org
Re: Moving /tmp to tmpfs is fine
[Ben Hutchings] > We should be thinking about implementing per-user temporary directories > and making sure that programs respect $TMPDIR. Yes, per-user temp directories is a good idea. Installing the libpam-tmpdir package enable this by default, and beside some problems with the root user (bad TMPDIR is inherited when I restart services using /etc/init.d/ scripts), it work well. Perhaps it should be extended to allow the directory to be below ~/ instead of below /tmp/. :) It make it very easy to spot the programs not respecting $TMPDIR. :) > (On Linux it's also possible to give each user a different /tmp > through mount namespaces. I'm not sure whether that's compatible > with historical use of /tmp by the X window system.) This sound a bit more scary, yes. -- Happy hacking Petter Reinholdtsen -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2fltxz0p18t@login2.uio.no
Re: Moving /tmp to tmpfs is fine
On 28/05/12 17:48, Thomas Goirand wrote: > On 05/28/2012 04:46 AM, Serge wrote: >> The truth is that tmpfs IS FASTER in some cases. The problem is that >> *nobody* can notice that on *real* applications. > Serge, I'm on your side of the discussion, but the above is simply > not truth. And by the way, that's not the issue. The issue is potential > breakage, which we want to avoid *at all costs*. > > Thomas > > I think this is a valid point. We should know what applications and workloads get a _measurable_ benefit by using tmpfs for /tmp instead of using a normal filesystem. If we are optimizing things for just a synthetic benchmark that does fsyncs on lot of small files then we are loosing the perspective of reality. We should have things on the table like the following to support the idea that tmpfs really gives any performance benefit in the day-to-day real-world-tasks of people and not only on synthetic benchmarks - Program X is a % faster when using tmpfs for /tmp - Compiling the Linux Kernel is a % faster when using tmpfs for /tmp - Task Z is a % faster when using tmpfs for /tmp - [...] Regards! -- ~~~ Carlos Alberto Lopez Perez http://neutrino.es Igalia - Free Software Engineeringhttp://www.igalia.com ~~~ signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs is fine
On 05/28/2012 04:46 AM, Serge wrote: > The truth is that tmpfs IS FASTER in some cases. The problem is that > *nobody* can notice that on *real* applications. Serge, I'm on your side of the discussion, but the above is simply not truth. And by the way, that's not the issue. The issue is potential breakage, which we want to avoid *at all costs*. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc39e64.10...@debian.org
Re: Moving /tmp to tmpfs is fine
On Mon, May 28, 2012 at 01:59:24AM +0100, Ben Hutchings wrote: I don't recall that being common practice on any multi-user Unix-like system I've used. (It's also not something a Windows user would expect, Well, it was back in my university days, and it still is where I work. Maybe „multi-user” is wrong, but telling other user that the ISO lies on my system in /tmp is much easier than to tell them a location in my $HOME and to make sure they can access ist. as they already get per-user temporary directories.) One of the first things I do after a Windows installation is to create c:\temp. ;-) Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: s...@fsing.rootsland.net | | Public Keys: http://fsing.rootsland.net/~stse/keys.html | signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs is fine
* Ben Hutchings [120527 17:25]: > Creating arbitrarily large temporary files outside the user's home > directory is generally going to be unreliable. The only thing more unreliable than that is creating arbitrary large file in user's home directory. If it is not supposed to be persistent data that is available on every node the user can log in, then it does not belong into the home directory (unless the user explicitly choose to set TMPDIR there). The home directory can be quite slow (because being remote, being encrypted, ...) and quite scarce (permanent storage on server discs with redudancy and backup strategies is not that cheap). Unless a program is explicitly told otherwise, temporary files belong to TMPDIR and if that is not set to /tmp. (or any subdirectory thereof the programs like to create). If that is too small, then it is too small. The admin is able to configure /tmp differently, the user is able to set TMPDIR differently. my 0.02: I personally think having tmpdir on /tmp might be a good default for new systems. If systems get changed to that from something else on upgrade without asking, I consider that quite an ugly bug. Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120528084732.ga4...@client.brlink.eu
Re: Moving /tmp to tmpfs is fine
+++ Henrique de Moraes Holschuh [2012-05-27 11:04 -0300]: > On Sat, 26 May 2012, Salvo Tomaselli wrote: > > > Or, it should get clever and not unpack everything. There are plenty of > > > software that are able to read into archives without extracting from > > > them. > > You can't do it for a .tar.gz or a .tar.bz and they are the most common > > kind > > of archive. > > Yes, you can. But it is slow (requires one full pass to get the file > catalog, and another to decompress file data), so you would do it only when > you expect it to be better than the alternative. > > Still, if mc will obey $TMPDIR, it is not at fault. Does it? Yes, it does. (I just checked). It even behaves well if the dir specified does not exist (complains, but starts anyway, gives errors if you try to open archives with no tmpdir, starts using it as soon as it appears). Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120528061543.gl11...@stoneboat.aleph1.co.uk
Re: Moving /tmp to tmpfs is fine
+++ Thorsten Glaser [2012-05-27 17:52 +]: > Wookey dixit: > > > >here's a case where a lot of space gets used in there: open a .ppt > >(powerpoint) file in libreoffice. The conversion involves writing a > >file in /tmp/ for every page/image. To open an image-heavy > >256Mb .ppt I have lying about here, generates 382MB of files in /tmp. > > Ouch. (But nobody said Staroffice/Openoffice/Libreoffice/whatsitsname > was light.) This sounds like another of these cases where the software > would benefit from changes _independent_ of the tmpfs setting. > >So I'm with serge that this can be a real-world problem. I'm > > I don’t disagree but still stand by that these are corner cases. How can 'opening a big .ppt I was just sent', be a corner case? That's the sort of thing 'normal people' do on a daily basis. And if we have any pretence of Debian being useful outside the people on this list we really should have defaults which mean it 'just works' (if your disk isn't full already). Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120528061122.gk11...@stoneboat.aleph1.co.uk
Re: Moving /tmp to tmpfs is fine
On Mon, 2012-05-28 at 10:40 +1000, Russell Coker wrote: > On Mon, 28 May 2012, Thomas Goirand wrote: > > On 05/27/2012 09:38 PM, Russell Coker wrote: > > > Sure it's easy for me to fix that when upgrading and when compared to all > > > the other things I have to do on an upgrade it's not much of a big deal. > > > > It's *not* easy, this involve init.d script foo ATM. See #674517. > > As noted in that bug report you can just edit /etc/default/rcS to make it not > use a tmpfs for /tmp. That is easy to fix. > > On Mon, 28 May 2012, Jon Dowland wrote: > > On Sun, May 27, 2012 at 04:25:30PM +0100, Ben Hutchings wrote: > > > We should be thinking about implementing per-user temporary directories > > > and making sure that programs respect $TMPDIR. (On Linux it's also > > > possible to give each user a different /tmp through mount namespaces. > > > I'm not sure whether that's compatible with historical use of /tmp by > > > the X window system.) > > > > Yes! This is a good idea for other reasons, too, including some disc > > encryption situations. Perhaps it's a candidate for a release goal for > > wheezy+1? Some scoping work is probably required. > > Using a bind mount to make /tmp/.X11-unix available to the logged in user > isn't going to be difficult. What is /tmp/.X0-lock used for? > > As for making it a release goal for wheezy+1, it can't be enabled by default > because usually the users expect to be able to share files via /tmp. I don't recall that being common practice on any multi-user Unix-like system I've used. (It's also not something a Windows user would expect, as they already get per-user temporary directories.) Ben. -- Ben Hutchings Teamwork is essential - it allows you to blame someone else. signature.asc Description: This is a digitally signed message part
Re: Moving /tmp to tmpfs is fine
On Mon, 28 May 2012, Thomas Goirand wrote: > On 05/27/2012 09:38 PM, Russell Coker wrote: > > Sure it's easy for me to fix that when upgrading and when compared to all > > the other things I have to do on an upgrade it's not much of a big deal. > > It's *not* easy, this involve init.d script foo ATM. See #674517. As noted in that bug report you can just edit /etc/default/rcS to make it not use a tmpfs for /tmp. That is easy to fix. On Mon, 28 May 2012, Jon Dowland wrote: > On Sun, May 27, 2012 at 04:25:30PM +0100, Ben Hutchings wrote: > > We should be thinking about implementing per-user temporary directories > > and making sure that programs respect $TMPDIR. (On Linux it's also > > possible to give each user a different /tmp through mount namespaces. > > I'm not sure whether that's compatible with historical use of /tmp by > > the X window system.) > > Yes! This is a good idea for other reasons, too, including some disc > encryption situations. Perhaps it's a candidate for a release goal for > wheezy+1? Some scoping work is probably required. Using a bind mount to make /tmp/.X11-unix available to the logged in user isn't going to be difficult. What is /tmp/.X0-lock used for? As for making it a release goal for wheezy+1, it can't be enabled by default because usually the users expect to be able to share files via /tmp. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205281040.51599.russ...@coker.com.au
Re: Moving /tmp to tmpfs is fine
Salvo Tomaselli wrote: >> Or, it should get clever and not unpack everything. There are plenty of >> software that are able to read into archives without extracting from >> them. > You can't do it for a .tar.gz or a .tar.bz and they are the most common kind > of archive. xz compression format supports dividing files into blocks that are seekable http://www.tukaani.org/xz/format.html tar format itself has limitations, here is some planning of new format http://duplicity.nongnu.org/new_format.html -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120527222518.GA19150@lisko
Re: Moving /tmp to tmpfs is fine
On Sun, May 27, 2012 at 4:43 PM, Thomas Goirand wrote:
> On 05/27/2012 02:52 AM, Mike Hommey wrote:
>> Or, it should get clever and not unpack everything. There are plenty of
>> software that are able to read into archives without extracting from
>> them. There are even fuse filesystems to do that if it doesn't want to
>> do it itself. Using a temporary directory, be it on disk or in RAM, is
>> *always* going to be a limitation.
> You may or may not be right. That's not the point. Things are what they
> are, and we have to deal with them. Unless you want to rewrite/patch:
> - Firefox
> - mc
> - mysql
> - {open,libre}office
gscan2pdf too, I use it to send my tax receipt and It crash during
scaning due to this issue (tmpfs full)
>
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/cae2spaahaspwj2kij9ndt09txqccdvfmwdddqyeavztqufh...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
2012/5/27 Iustin Pop wrote: > There's a difference between "tmpfs is bad" and "the defaults for tmpfs > are bad". First, I'm not saying that tmpfs is just bad. It's GOOD. For some cases. I use it myself when I need to be sure that (having enough RAM) some of my *large* files will never leave the cache and will *read* really fast even when not used for days. It's not about "tmpfs is bad". It's about "/tmp on tmpfs is bad". And if "/tmp on tmpfs is bad", it does mean that "defaults for tmpfs are bad", doesn't it? > There are benefits, but your broken benchmarks don't show it. Possible. That's why in almost every email I'm asking for a better test, better usecase, some popular applications, anything, proving it's good. But still... > Your tests are wrong, as Adam Borowski very nicely explained. My "test" was `time tar xf ~/linux-3.4.tar.bz2`. It's not "wrong", it's probably the most real test suggested so far. Aren't you using tar/bzip2? It's a much better test than some artificial bash/perl-scripts that nobody use in real-life. And we're still talking about real-life default settings, I hope. Those bash snippets I wrote were just to check about fsync(). It's stupid to change defaults because some useless bash script works faster. Imagine that I wrote an application that works faster on reiserfs than on ext*fs. Will you change debian *default* root filesystem to reiserfs because of my own application that nobody else uses? The truth is that tmpfs IS FASTER in some cases. The problem is that *nobody* can notice that on *real* applications. So in some artificial world on another planet /tmp on tmpfs could be a good idea, but we're in the real world on Earth, aren't we? >> I don't dismiss them. But we talk about *defaults*. And I don't know >> any real applications, heavily fsync()ing files in /tmp, that people >> are expected to use by default. Can you name some? > Which people? You can't overgeneralise. Ok, honestly, I don't know ANY popular applications, heavily fsync()ing files in /tmp. Can you name some? Otherwise what's the reason to optimize for fsync() if noone uses it for /tmp? > But I don't appreciate the fact that, in your overzealous attitude, you: > - come up with faulty benchmarks, which show that you misunderstand what > the bottlenecks are > - make assumptions that people are using tmpfs because they are ignorant > - claim that people are using tmpfs only because they have overpowered > hardware My guess about people using tmpfs is based on the fact that during last few days of this discussion and a few monthes of previous discussions I've read through nobody had finally said why using /tmp on tmpfs is good. Everybody are trying to say why it's "not that bad", but *nobody* explains why it's good. This is the first thread where we're at least trying to do that (first benchmarks appeared). And, by the way, I see nothing bad in people being ignorant. I am ignorant in some things. It's impossible to know everything in the world. That's why I ask others to help me and find out why /tmp on tmpfs is good. Or, if we won't find it, disable it by default. > Honestly, other people in this thread have made the point against tmpfs > much better than you That's because I'm NOT trying to find why /tmp on tmpfs is bad. It's easy and obvious. Instead I'm trying to find why it's a good default. Is it? Why? Is it good just because it's "not that bad"? > I will stop replying to this thread, because I don't have much to add; > there are pros and cons to both solutions Then, can you, please, mail me directly, and name the pros. I'm trying to collect all the points. And I still miss yours. I understand you believe it's not bad for you. Ok. But why /tmp on tmpfs is good for you? -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caoveneqyifbjwv4z1zn7to0x9x+urd6cggaxkvcod6gs-q6...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
On Sun, May 27, 2012 at 04:25:30PM +0100, Ben Hutchings wrote: > We should be thinking about implementing per-user temporary directories > and making sure that programs respect $TMPDIR. (On Linux it's also > possible to give each user a different /tmp through mount namespaces. > I'm not sure whether that's compatible with historical use of /tmp by > the X window system.) Yes! This is a good idea for other reasons, too, including some disc encryption situations. Perhaps it's a candidate for a release goal for wheezy+1? Some scoping work is probably required. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120527193743.GA3678@debian
Re: Moving /tmp to tmpfs is fine
Ben Hutchings wrote: > As will /tmp on a small root partition. > As will a small dedicated /tmp partition. The differences between these cases and forcing tmpfs by default is that in the above cases, the person who installed the system chose those partition sizes. They are therefore responsible for the breakage they caused, and they can reason about this: "I told it to make / 200 mb, so of course it can't write a CD iso there", and fix it. However, the user who slaps a 2 terabyte disk in, and puts Debian on it, has not made a choice that explains why Debian fails to burn a CD due to having ignored all that disk space behind their back. > We should be thinking about implementing per-user temporary directories > and making sure that programs respect $TMPDIR. Absolutely, but it's orthagonal to breaking previously sane defaults in the size of /tmp. -- see shy jo signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs is fine
On 27/05/12 17:47, Thomas Goirand wrote: > On 05/27/2012 11:25 PM, Ben Hutchings wrote: >> As will /tmp on a small root partition. >> As will a small dedicated /tmp partition. >> > Why taking just bad configurations as counter arguments? > Do you know it is as well possible to have enough space > in your /tmp? :) > I agree. Remember that we are not talking about if "tmpfs on /tmp" is good or bad, but about if it is a good default. Actually the default in squeeze is to have /tmp _as_ _large_ as your disk * All files in one partition (recommended for new users) [1] And since we are talking about defaults, I think that having "tmpfs on /tmp" as default for Wheezy is a very bad idea. If you know what tmpfs is, and you think it can improve things on your machine: then just turn it on!. But please: don't make it a default, since many users don't know what is tmpfs or a partition. And if they are going to find all sort of problems because we did a bad choice with this then is a problem for Debian and its users. Regards! [1] http://www.linuxbsdos.com/2011/02/15/debian-6-installation-and-disk-partitioning-guide/ -- ~~~ Carlos Alberto Lopez Perez http://neutrino.es Igalia - Free Software Engineeringhttp://www.igalia.com ~~~ signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs is fine
Thorsten Glaser wrote: > >On 25/05/2012 18:20, Salvo Tomaselli wrote: > >> Double-click on a .tar causes it to be unpacked in /tmp/something. > >> I suppose a lot of not so skilled users do that instead of tar -xf > > > >That doesn't seem to happen with file-roller. Perhaps you need to file a > >bug > Hm. mc does put things into /tmp as does debdiff, but the latter > at least honours TMPDIR (and --no-unpack-tarballs). > > But in the very most cases, I *do* want them to be extracted in > /tmp as they “usually” fit. So I’d rather have a heuristic put > into the file manager whether to set TMPDIR before calling the > extraction utility (or which tmpdir to use if it designates the > extraction place by itself). mc maintainers, are you listening? If you want to reach package maintainer, you have to mail them specifically. They don't have to be subscribed to debian-devel. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120527180441.GA12708@lisko
Re: Moving /tmp to tmpfs is fine
Wookey dixit: >But there is this issue of the way its vfs does temporay unpacking in >/tmp. That makes sense in the 'this is temporary, it should go away on >reboot' sense, but some big files will use up a lot of ram when /tmp >is tmpfs. > >I don't know what the right thing to do about this is, but the 'set >TMPDIR' sugestion is useless IMHO. When I start mc I have no idea what >things will be unpacked under the VFS over the next days/weeks. I Yes. What I was trying to say was: set TMPDIR for many operations, but file managers, such as mc, should be patched to apply heuristics to determine whether to use /tmp or /var/tmp. >don't want _everything_ to go in /var/tmp and not get cleaned up >automatically (is there cron job that does this for old files?) I think there is, but may be confusing with BSD, from which I know for sure /tmp is cleaned at reboot and, daily, files older than seven days. (Note that we see another benefit of tmpfs for /tmp and for *most* files here: when mc segfaults again, its tempfiles will not linger around long when in /tmp on tmpfs…) >Perhaps mc should get clever and change TMPDIR itself on the fly for >large files, but I don't know how practical that is. Yes! >here's a case where a lot of space gets used in there: open a .ppt >(powerpoint) file in libreoffice. The conversion involves writing a >file in /tmp/ for every page/image. To open an image-heavy >256Mb .ppt I have lying about here, generates 382MB of files in /tmp. Ouch. (But nobody said Staroffice/Openoffice/Libreoffice/whatsitsname was light.) This sounds like another of these cases where the software would benefit from changes _independent_ of the tmpfs setting. (It could for example do that only for the first few pages, doing others as the timeline progresses.) >So I'm with serge that this can be a real-world problem. I'm I don’t disagree but still stand by that these are corner cases. bye, //mirabilos -- 13:37⎜«Natureshadow» Deep inside, I hate mirabilos. I mean, he's a good guy. But he's always right! In every fsckin' situation, he's right. Even with his deeply perverted taste in software and borked ambition towards broken OSes - in the end, he's damn right about it :(! […] works in mksh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205271749020.24...@herc.mirbsd.org
Re: Moving /tmp to tmpfs is fine
On 05/27/2012 11:25 PM, Ben Hutchings wrote: > As will /tmp on a small root partition. > As will a small dedicated /tmp partition. > Why taking just bad configurations as counter arguments? Do you know it is as well possible to have enough space in your /tmp? :) Cheers, Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc24c96.2040...@debian.org
Re: Moving /tmp to tmpfs is fine
On 05/27/2012 09:38 PM, Russell Coker wrote: > Sure it's easy for me to fix that when upgrading and when compared to all the > other things I have to do on an upgrade it's not much of a big deal. It's *not* easy, this involve init.d script foo ATM. See #674517. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc24a6c.70...@debian.org
Re: Moving /tmp to tmpfs is fine
On Sun, 2012-05-27 at 22:43 +0800, Thomas Goirand wrote:
> On 05/27/2012 02:52 AM, Mike Hommey wrote:
> > Or, it should get clever and not unpack everything. There are plenty of
> > software that are able to read into archives without extracting from
> > them. There are even fuse filesystems to do that if it doesn't want to
> > do it itself. Using a temporary directory, be it on disk or in RAM, is
> > *always* going to be a limitation.
> You may or may not be right. That's not the point. Things are what they
> are, and we have to deal with them. Unless you want to rewrite/patch:
> - Firefox
> - mc
> - mysql
> - {open,libre}office
> - ...
>
> then /tmp using tmpfs *will* lead to issues that many wont understand.
As will /tmp on a small root partition.
As will a small dedicated /tmp partition.
Creating arbitrarily large temporary files outside the user's home
directory is generally going to be unreliable. A shared /tmp also
results in various security problems (mostly mitigated by link
restrictions) and privacy problems (I can see the names of the files
your browser downloaded).
We should be thinking about implementing per-user temporary directories
and making sure that programs respect $TMPDIR. (On Linux it's also
possible to give each user a different /tmp through mount namespaces.
I'm not sure whether that's compatible with historical use of /tmp by
the X window system.)
Ben.
--
Ben Hutchings
The obvious mathematical breakthrough [to break modern encryption] would be
development of an easy way to factor large prime numbers. - Bill Gates
signature.asc
Description: This is a digitally signed message part
Re: Moving /tmp to tmpfs is fine
On 05/27/2012 02:52 AM, Mike Hommey wrote:
> Or, it should get clever and not unpack everything. There are plenty of
> software that are able to read into archives without extracting from
> them. There are even fuse filesystems to do that if it doesn't want to
> do it itself. Using a temporary directory, be it on disk or in RAM, is
> *always* going to be a limitation.
You may or may not be right. That's not the point. Things are what they
are, and we have to deal with them. Unless you want to rewrite/patch:
- Firefox
- mc
- mysql
- {open,libre}office
- ...
then /tmp using tmpfs *will* lead to issues that many wont understand.
Thomas
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4fc23d8d.9020...@debian.org
Re: Moving /tmp to tmpfs is fine
On 05/27/2012 01:59 AM, Wookey wrote: > here's a case where a lot of space gets used in there: open a .ppt > (powerpoint) file in libreoffice. The conversion involves writing a > file in /tmp/ for every page/image. To open an image-heavy > 256Mb .ppt I have lying about here, generates 382MB of files in /tmp. > Oh, that's right! I forgot about this one. I had the issue with openoffice once, and my 1GB /tmp partition wasn't enough. It filled more than 2GB of crap, in fact, and if this was on a tmpfs, then it wouldn't have worked at all (eg: I wouldn't have had enough RAM at the time). It took me a long time to understand what was going on. If it was someone else, like my mother or my wife (yes, they do use Debian :)), of course, they would never understand it. Thanks for reminding me this one which we have to add to the big list of heavy users of big-files in /tmp. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc23cad.7040...@debian.org
Re: Moving /tmp to tmpfs is fine
On Sat, 26 May 2012, Salvo Tomaselli wrote: > > Or, it should get clever and not unpack everything. There are plenty of > > software that are able to read into archives without extracting from > > them. > You can't do it for a .tar.gz or a .tar.bz and they are the most common kind > of archive. Yes, you can. But it is slow (requires one full pass to get the file catalog, and another to decompress file data), so you would do it only when you expect it to be better than the alternative. Still, if mc will obey $TMPDIR, it is not at fault. Does it? -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120527140401.gc29...@khazad-dum.debian.net
Re: Moving /tmp to tmpfs is fine
On Sun, 27 May 2012, Iustin Pop wrote: > There's a difference between "tmpfs is bad" and "the defaults for tmpfs > are bad". The new defaults don't seem good when they are suddenly applied on upgrade. My workstation unexpectedly went from having 2G of free space on the root filesystem for /tmp to 600M of tmpfs. 600M is almost filled by two TED talks so with my habits of downloading multiple video files that was never going to work. I think it would be a great feature to have the Debian installer give an option of a tmpfs for /tmp. I think it would be quite OK to have it default to tmpfs on /tmp but give the user the option of doing otherwise. But having it just default to tmpfs and change existing systems on upgrade doesn't seem that good. This discussion has demonstrated that there are more than a few good reasons to support both using tmpfs and not using tmpfs for /tmp. But I can't think of any good reason for changing a working system, all of my systems running Squeeze which should have a tmpfs on /tmp (IMHO - and I'm really the one who knows) already have it. There is no need for a change when I upgrade. Sure it's easy for me to fix that when upgrading and when compared to all the other things I have to do on an upgrade it's not much of a big deal. But it would still be good to not be surprised. For installing new systems I don't think it matters much what the default is, whatever is chosen will be good for some, bad for others, and probably not matter to most people. But making it easy to change is a good thing. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205272338.19181.russ...@coker.com.au
Re: Moving /tmp to tmpfs is fine
On Sun, May 27, 2012 at 05:39:21AM +0300, Serge wrote: > 2012/5/25 Iustin Pop wrote: > > > And no, "I really can't think of any popular application" is not a valid > > discussion point. > > But there're already popular applications and usecases that break because > of that. It can render the system unstable because of heavy swap usage. > So there must be some strong point to still use it despite those problems. > There must be some very popular application, that do not break because > of that feature and even becomes better. There's a difference between "tmpfs is bad" and "the defaults for tmpfs are bad". > Otherwise, if this feature causes problems to some applications and no > benefits to others, what's the point in using it? There are benefits, but your broken benchmarks don't show it. > > This is plain wrong. NO benefits for tmpfs? "just works somehow"? > > Ok, I must be missing some obvious benefit. Please, help me and name it. > But real one not theoretical. Some real (and popular, since we're talking > about defaults) application that becomes faster (or better in any other > way) because of /tmp being on tmpfs. All the tests showed tmpfs being no > better than ext3 so far. Your tests are wrong, as Adam Borowski very nicely explained. > > you only look at _your_ use case and dismiss all others, or that you > > don't understand the different behaviours of fsync() (with enough memory, > > that is) on tmpfs, HDDs and SSDs. > > I don't dismiss them. But we talk about *defaults*. And I don't know > any real applications, heavily fsync()ing files in /tmp, that people are > expected to use by default. Can you name some? Which people? You can't overgeneralise. I agree that the default sizes of tmpfs are likely wrong. But that doesn't mean, as you claim, that tmpfs itself is wrong. > > iustin, happily using /tmp on tmpfs since many, many years ago > > Heh... A lot of people use it. I guess most of them have seen "/tmp", then > they were seing "tmpfs", and decided that "tmpfs is the fs for /tmp", it > seemed natural to them. They never really thought, whether it's good or > bad idea, or that there may be some better ideas. It was just natural to > use it. I appreciate this attack. It helps your point very much to paint people who argue for tmpfs as clueless people. > And when I say, "hey, that's a bad idea", I hurt them (I'm sorry for that). > They start arguing that "it's not that bad as you say, look, not everything > is broken, many programs still work". They can't say why it's better than > using disk because they never though about that. It was just "natural"... Serge, I very much appreciate the fact that you're trying to make a better experience for Debian users. But I don't appreciate the fact that, in your overzealous attitude, you: - come up with faulty benchmarks, which show that you misunderstand what the bottlenecks are - make assumptions that people are using tmpfs because they are ignorant - claim that people are using tmpfs only because they have overpowered hardware - etc. Honestly, other people in this thread have made the point against tmpfs much better than you; I would suggest you tone down a bit your position, and stop assuming ignorance on other's people part. I will stop replying to this thread, because I don't have much to add; there are pros and cons to both solutions, but I personally I'm still surprised that people don't see the advantage of tmpfs for not underpowered memory cases. regards, iustin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120527132036.ga19...@teal.hq.k1024.org
Re: Moving /tmp to tmpfs is fine
2012/5/25 Iustin Pop wrote: > And no, "I really can't think of any popular application" is not a valid > discussion point. But there're already popular applications and usecases that break because of that. It can render the system unstable because of heavy swap usage. So there must be some strong point to still use it despite those problems. There must be some very popular application, that do not break because of that feature and even becomes better. Otherwise, if this feature causes problems to some applications and no benefits to others, what's the point in using it? > This is plain wrong. NO benefits for tmpfs? "just works somehow"? Ok, I must be missing some obvious benefit. Please, help me and name it. But real one not theoretical. Some real (and popular, since we're talking about defaults) application that becomes faster (or better in any other way) because of /tmp being on tmpfs. All the tests showed tmpfs being no better than ext3 so far. > you only look at _your_ use case and dismiss all others, or that you > don't understand the different behaviours of fsync() (with enough memory, > that is) on tmpfs, HDDs and SSDs. I don't dismiss them. But we talk about *defaults*. And I don't know any real applications, heavily fsync()ing files in /tmp, that people are expected to use by default. Can you name some? > iustin, happily using /tmp on tmpfs since many, many years ago Heh... A lot of people use it. I guess most of them have seen "/tmp", then they were seing "tmpfs", and decided that "tmpfs is the fs for /tmp", it seemed natural to them. They never really thought, whether it's good or bad idea, or that there may be some better ideas. It was just natural to use it. And when I say, "hey, that's a bad idea", I hurt them (I'm sorry for that). They start arguing that "it's not that bad as you say, look, not everything is broken, many programs still work". They can't say why it's better than using disk because they never though about that. It was just "natural"... That's just my guess, of course. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caoveneom+nfemjxs61xqekbbt0ui5_epw0yl9t930ratvhe...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
> Or, it should get clever and not unpack everything. There are plenty of > software that are able to read into archives without extracting from > them. You can't do it for a .tar.gz or a .tar.bz and they are the most common kind of archive. -- Salvo Tomaselli -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205262113.26922.tipos...@tiscali.it
Re: Moving /tmp to tmpfs is fine
On Sat, May 26, 2012 at 06:59:35PM +0100, Wookey wrote: > I hesitate to prolong this thread further, but I do have a couple of > data points. (and couldn't let Neil's nonsense go). > > +++ Neil Williams [2012-05-25 16:15 +0100]: > > > So instead of fixing the defaults you suggest everybody to drop the > > > programs they use (mc, firefox, mysql)? ;) > > > > On machines which don't have the resources for those programs, yes. > > There are alternatives out there - change to a less hungry program or > > expand the hardware. > > The idea that there are machines without the resources for mc is > silly. It's a very low-resource console-based program. > > But there is this issue of the way its vfs does temporay unpacking in > /tmp. That makes sense in the 'this is temporary, it should go away on > reboot' sense, but some big files will use up a lot of ram when /tmp > is tmpfs. > > I don't know what the right thing to do about this is, but the 'set > TMPDIR' sugestion is useless IMHO. When I start mc I have no idea what > things will be unpacked under the VFS over the next days/weeks. I > don't want _everything_ to go in /var/tmp and not get cleaned up > automatically (is there cron job that does this for old files?) > > Perhaps mc should get clever and change TMPDIR itself on the fly for > large files, but I don't know how practical that is. Or, it should get clever and not unpack everything. There are plenty of software that are able to read into archives without extracting from them. There are even fuse filesystems to do that if it doesn't want to do it itself. Using a temporary directory, be it on disk or in RAM, is *always* going to be a limitation. As a matter of fact, i remember some virtualization solution (vserver or lxc, i think) that gave me a 16MB /tmp by default. It doesn't matter if it's on disk or in RAM at this point. If mc is going to try to unpack a kernel archive in there, it's going to blatantly fail. Yet, there's no excuse for it to not do better. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120526185205.ga24...@glandium.org
Re: Moving /tmp to tmpfs is fine
I hesitate to prolong this thread further, but I do have a couple of data points. (and couldn't let Neil's nonsense go). +++ Neil Williams [2012-05-25 16:15 +0100]: > > So instead of fixing the defaults you suggest everybody to drop the > > programs they use (mc, firefox, mysql)? ;) > > On machines which don't have the resources for those programs, yes. > There are alternatives out there - change to a less hungry program or > expand the hardware. The idea that there are machines without the resources for mc is silly. It's a very low-resource console-based program. But there is this issue of the way its vfs does temporay unpacking in /tmp. That makes sense in the 'this is temporary, it should go away on reboot' sense, but some big files will use up a lot of ram when /tmp is tmpfs. I don't know what the right thing to do about this is, but the 'set TMPDIR' sugestion is useless IMHO. When I start mc I have no idea what things will be unpacked under the VFS over the next days/weeks. I don't want _everything_ to go in /var/tmp and not get cleaned up automatically (is there cron job that does this for old files?) Perhaps mc should get clever and change TMPDIR itself on the fly for large files, but I don't know how practical that is. I only have 2G in this machine and astonishing slowdown due to thrashing is quite common after a few days work (generally firefoxes fault). I do begrudge /tmp (and thus tmpfs) having more than a few MB of ram. hmm. I just checked some stats: after looking inside a 45MB .deb files in mc: tmpfs382M 212K 382M 1% /tmp so it's not unpacking it there any more even though MC_TMPDIR=/tmp/mc-wookey perhaps this issue is fixed at least for this app? here's a case where a lot of space gets used in there: open a .ppt (powerpoint) file in libreoffice. The conversion involves writing a file in /tmp/ for every page/image. To open an image-heavy 256Mb .ppt I have lying about here, generates 382MB of files in /tmp. I tried to do this live when someone was having trouble with a presentation - it took over 20mins to open this file due to thrashing and I was unable to save the presenation from being something of a disaster. /tmp being a tmpfs presumably contributed to that failure (It just opened fine in about 2mins on the same machine with less memory pressure). And of course I didn't set 'TMPDIR' beforehand a) because all I did was double-click on the file in the filer, which ran libreoffice impress and b) because I had no idea that it would general hundreds of megs of files in /tmp. (I found out afterwards). So I'm with serge that this can be a real-world problem. I'm not claiming that the only solution is a real-disk /tmp, because I agree with those who do see advantages in /tmp as tmpfs so long as /tmp stays fairly small. > No problem with tmpfs being in RAM, it's all about being rational in > the selection of packages. 'firefox' is a perfectly rational choice of package on this laptop of 2G ram. mc is a rational choice on any machine. You can't solve the issue of 'unpacking huge tarballs/debs/whatevers' in /tmp' by telling people to use different packages. Something else is needed. It is possible that that something else already exists, but I must admit to now being a little confused about what actually happens as this thread contains conflicting info. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120526175935.ge11...@stoneboat.aleph1.co.uk
Re: Moving /tmp to tmpfs is fine
On Fri, May 25, 2012 at 08:14:10PM +0300, Serge wrote: > 2012/5/25 Neil Williams wrote: > > Different hardware -> different software selection. > > I don't understand your point. I could understand it if we were choosing > among benefits that most users get from /tmp being on disk and /tmp being > on tmpfs. But there're NO benefits in having /tmp on tmpfs. It works (not > works better, just works somehow) only on systems with a lot of RAM. This is plain wrong. NO benefits for tmpfs? "just works somehow"? Whatever other arguments you had, the statement above tells me you only look at _your_ use case and dismiss all others, or that you don't understand the different behaviours of fsync() (with enough memory, that is) on tmpfs, HDDs and SSDs. And no, "I really can't think of any popular application" is not a valid discussion point. iustin, happily using /tmp on tmpfs since many, many years ago, and configuring it as such on all Debian machines he installs (of various roles). -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120525202638.ga19...@teal.hq.k1024.org
Re: Moving /tmp to tmpfs is fine
2012/5/25 Neil Williams wrote: > Do you not use swap? I use it for suspend-to-disk. > Yes you lose functionality but functionality takes up resources, so > something has to give. Which functionality will I lose by placing /tmp on the real disk? > The vast majority of systems have large amounts of swap, so tmpfs never > runs out until the swap space is full - it just gets very, very much > slower. Right, system will become unusable and user will press Reset button far before that. >> Assuming you've set your tmpfs size to 20% you need 2.5GB memory just >> to "temporarily" unpack kernel sources and check for some files. > And? The machines I use to unpack and repack kernel packages handle that > quite nicely. Since we're talking about default settings, you assume default debian system to have at least 2.5GB just to be able to unpack kernels? > Different hardware -> different software selection. I don't understand your point. I could understand it if we were choosing among benefits that most users get from /tmp being on disk and /tmp being on tmpfs. But there're NO benefits in having /tmp on tmpfs. It works (not works better, just works somehow) only on systems with a lot of RAM. And nobody still named any popular programs, that can definitely benefit from that. While there're many programs that either break or may render system unstable. Yes, /tmp on tmpfs works in many cases. But /tmp on disk works always. Why would we select the worst of these two options? I don't understand, do you suggest to break some applications and make system less stable for nothing? What's a good part? -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOVenEr24CyHFALpn5J0mnfqLk+S3ovHqL+=bMR6qr-rVH=9...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
On Fri, 25 May 2012 15:25:58 +0300 Serge wrote: > 2012/5/25 Neil Williams wrote: > > > You cannot expect to mix those two worlds and for things to "just > > work". > > Easy. Let's leave /tmp on a real disk and both world will "just work". Do you not use swap? > > If program A is too resource-hungry, find (or write) program B. > > Or fix the program A, right? And here we go... By default the program > Debian is too memory-hungry (with large tmpfs) or breaking apps (with > small tmpfs). Let's fix that? ;) Write program C which does only the bits of what program A does but does it by using a lot less resources (generally there's no point differentiating between machines with low RAM and low storage space). e.g. netsurf in place of iceweasel. GPE instead of XFCE/Gnome. Yes you lose functionality but functionality takes up resources, so something has to give. > > The default is fine and sane but no default will ever satisfy every > > possible device. Low memory devices have many many more problems than > > just where /tmp is mounted. > > Every system becomes "Low memory" with these defaults. Huh? Not true. The vast majority of systems have large amounts of swap, so tmpfs never runs out until the swap space is full - it just gets very, very much slower. > Assuming you've > set your tmpfs size to 20% you need 2.5GB memory just to "temporarily" > unpack kernel sources and check for some files. And? The machines I use to unpack and repack kernel packages handle that quite nicely. > > That does NOT mean that Debian should change the default just to suit > > low memory devices. [...] we just have to be careful what applications > > we use. > > So instead of fixing the defaults you suggest everybody to drop the > programs they use (mc, firefox, mysql)? ;) On machines which don't have the resources for those programs, yes. There are alternatives out there - change to a less hungry program or expand the hardware. I write packages for both types of machines - I use powerful servers with lots of RAM and lots and lots of disc space for the repetitive tasks of creating smaller packages which can be more easily handled by low resource devices which have no swap, less than 1Gb of SSD and only 512Mb RAM. Then I write lots of new code on nice shiny workstations with lots of RAM and lots of disc space and deploy those packages on the units which gives a full user interface environment using (currently) ~21% of the available 512MB of RAM. rootfs1020M 757M 264M 75% / tmpfs 62M 0 62M 0% /lib/init/rw No problem with tmpfs being in RAM, it's all about being rational in the selection of packages. That way, the units can run on minimal power for days when the original desktop would drain the UPS in 10 minutes. Different hardware -> different software selection. When a package requires too many resources for a particular device, you simply choose another package, write another package or expand the hardware. As the proverb goes, you can't fit a gallon into a pint pot. http://www.emdebian.org/ -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpKlMn11btUk.pgp Description: PGP signature
Re: Moving /tmp to tmpfs is fine
2012/5/25 Neil Williams wrote: > You cannot expect to mix those two worlds and for things to "just > work". Easy. Let's leave /tmp on a real disk and both world will "just work". > If program A is too resource-hungry, find (or write) program B. Or fix the program A, right? And here we go... By default the program Debian is too memory-hungry (with large tmpfs) or breaking apps (with small tmpfs). Let's fix that? ;) > The default is fine and sane but no default will ever satisfy every > possible device. Low memory devices have many many more problems than > just where /tmp is mounted. Every system becomes "Low memory" with these defaults. Assuming you've set your tmpfs size to 20% you need 2.5GB memory just to "temporarily" unpack kernel sources and check for some files. Or it's supposed to be unpacked not in /tmp? And other programs should not be using /tmp to write large files? Then what *real* programs should use /tmp now? Is it useless for popular programs? > That does NOT mean that Debian should change the default just to suit > low memory devices. [...] we just have to be careful what applications > we use. So instead of fixing the defaults you suggest everybody to drop the programs they use (mc, firefox, mysql)? ;) -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOVenEpEOyAyOVKszAJvdJg2Urf-LqvNzC1=kvvrs04kcne...@mail.gmail.com
Re: Moving /tmp to tmpfs is fine
Chow Loong Jin dixit: >On 25/05/2012 18:20, Salvo Tomaselli wrote: >> Double-click on a .tar causes it to be unpacked in /tmp/something. >> I suppose a lot of not so skilled users do that instead of tar -xf > >That doesn't seem to happen with file-roller. Perhaps you need to file a bug Hm. mc does put things into /tmp as does debdiff, but the latter at least honours TMPDIR (and --no-unpack-tarballs). But in the very most cases, I *do* want them to be extracted in /tmp as they “usually” fit. So I’d rather have a heuristic put into the file manager whether to set TMPDIR before calling the extraction utility (or which tmpdir to use if it designates the extraction place by itself). mc maintainers, are you listening? bye, //mirabilos -- "Using Lynx is like wearing a really good pair of shades: cuts out the glare and harmful UV (ultra-vanity), and you feel so-o-o COOL." -- Henry Nelson, March 1999 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205251155530.7...@herc.mirbsd.org
Re: Moving /tmp to tmpfs is fine
On 25/05/2012 18:20, Salvo Tomaselli wrote: > Double-click on a .tar causes it to be unpacked in /tmp/something. > I suppose a lot of not so skilled users do that instead of tar -xf That doesn't seem to happen with file-roller. Perhaps you need to file a bug with your graphical archiver program. -- Kind regards, Loong Jin signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs is fine
> Doing that on inferior hardware is just plain stupid. If you have > plenty of > disk space, just unpack the tar archive. Double-click on a .tar causes it to be unpacked in /tmp/something. I suppose a lot of not so skilled users do that instead of tar -xf > And those with lots of RAM but not so much disk space (SD card or USB > driver or > even with no hard drive at all)? Can you link me one such device? Nothing pops in my mind. And how widespread they are? Normal desktop/laptop configurations have at least 100 times more disk space than ram. For servers.. it depends on the taks they are supposed to do, but usually servers have an administrator who is more or less aware of what he is doing. > There's no solution that works for everyone in all situations. True. But experts can change the defaults more effectively than non experts. > However, tmpfs at least works for many of them. If you KNOW > that this default does not fit your use-case, why don't you simply > change the configuration? The point is that the desktop user is often clueless and doesn't know it. He will just notice the problem and will not have a clue of how to solve it. So I would suggest that defaulting to a more safe configuration would be better, and of course those who want tmpfs can always change the default. But those who don't even know about what /tmp is would not end up with something that doesn't work so well on their hardware. Bye -- Salvo Tomaselli -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205251220.11062.tipos...@tiscali.it
Re: Moving /tmp to tmpfs is fine
Am 2012-05-25 11:19, schrieb Salvo Tomaselli: It's beginning to sound like your particular machines need either more RAM or to use a different temporary location which is on a permanent location. Just add some rules to clean it all up at reboot. Perhaps there are a couple of thousand users with the same use case, I don't know if it is the case but should be investigated rather than discarded. That does NOT mean that Debian should change the default just to suit low memory devices. So let's put minimum requirements unnecessarily high so a few people with super expensive laptops can have a 0.3μs speedup? (And people with cheaper hardware might never find out why the hell "linux" freezes if they click on a large tar archive). Doing that on inferior hardware is just plain stupid. If you have plenty of disk space, just unpack the tar archive. No. The default is fine and sane but no default will ever satisfy every possible device. Low memory devices have many many more problems than just where /tmp is mounted. But with tmpfs on disk, more devices would work by default (the ones with a lot of memory and disk, and the ones without much memory but with disk space). And those with lots of RAM but not so much disk space (SD card or USB driver or even with no hard drive at all)? There's no solution that works for everyone in all situations. However, tmpfs at least works for many of them. If you KNOW that this default does not fit your use-case, why don't you simply change the configuration? HS -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/d77602e05a6b6ce8cd0072a717bf8...@mail.hendrik-sattler.de
Re: Moving /tmp to tmpfs is fine
> It's beginning to sound like your particular machines need either more > RAM or to use a different temporary location which is on a permanent > location. Just add some rules to clean it all up at reboot. Perhaps there are a couple of thousand users with the same use case, I don't know if it is the case but should be investigated rather than discarded. > That does NOT mean that Debian should change the default just to suit > low memory devices. So let's put minimum requirements unnecessarily high so a few people with super expensive laptops can have a 0.3μs speedup? (And people with cheaper hardware might never find out why the hell "linux" freezes if they click on a large tar archive). > No. The default is fine and sane but no default will ever satisfy every > possible device. Low memory devices have many many more problems than > just where /tmp is mounted. But with tmpfs on disk, more devices would work by default (the ones with a lot of memory and disk, and the ones without much memory but with disk space). Bye -- Salvo Tomaselli -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205251119.15851.tipos...@tiscali.it
Re: Moving /tmp to tmpfs is fine
On Fri, 25 May 2012 02:22:24 +0300 Serge wrote: > What's a temporary file? Really, why would applications temporarily store > its data in a file? They do that to *free some memory*. Placing those files > back to memory renders the whole process of writing the file useless. Most of the time when I'm writing stuff to use /tmp (or $TMPDIR wherever that is), I'm using it only because I want to unpack something which is already a file into a temporary location which is fast and easy to remove without permission problems or having to setup anything in advance. RAM is ideal for that. $TMPDIR is ideal when a program needs to download something (or often a collection of somethings) and then needs to inspect them, checksum them, analyse, process and report on stuff about them and then throw them away and move on to the next collection of somethings. Now that may include processing them to upload them to somewhere else but still the local temporary copy needs to be cleaned up too. $TMPDIR is also important for control sockets and other clever pipe type stuff. > If the files are small and can stay in memory why would application save it > to file? Because it started out as a file and the program just wants to use some of the stuff inside the file? > Moving /tmp to tmpfs is effectively the same as suggesting to delete /tmp, > because there's no use for it as a temporary files storage any more. Not true. Temporary file storage is exactly why I need /tmp and temporary files don't get much more temporary than when they only exist in RAM. Having tmpfs to make RAM look like a filesystem is ideal. > FHS > === > Filesystem Hierarchy Standard defines two directories for temporary files: > /var/tmp — for files that should be preserved between reboots > /tmp — for files that should not be preserved between reboots > It's simple and clear. Yes and it is just what my programs need - space for files which need to be retrieved as files, processed as files but then blown away trivially. It means that the programs need to be run on devices with reasonable amounts of RAM - I don't really care. The programs need to be run on devices with a lot of disc space too, a fast processor and a fast network connection. That's life. I also write programs which have to work on tiny SSD drives, in minimal amounts of RAM and no network at all. You cannot expect to mix those two worlds and for things to "just work". If program A is too resource-hungry, find (or write) program B. > Since it's only reasonable to store large data sets in temporary files, > standard sets no size limits for these files. So if application's author > had actually read FHS he should expect these directories to handle > large files. Temporary files can be large files. Large files may only need to exist for a few seconds, it doesn't matter. Match the program to the resources available on that device and if one truly doesn't exist, write it. > Let's check the real world and see what applications actually use /tmp. > When you copy files in `mc` they're copied over /tmp/mc-username (to > handle some complex cases, like copying from inside iso-image to ssh). > When you click on a file in Firefox and select "Open with", Firefox stores > that file in /tmp. You cannot assume these files to be small. When you > watch large videos, adobe flash stores downloaded part of it as something > like /tmp/FlashXXG49VWF. Archive managers may unpack archives to /tmp. > CD burners store iso-files there. Image processing software was already > mentioned in this list. It's beginning to sound like your particular machines need either more RAM or to use a different temporary location which is on a permanent location. Just add some rules to clean it all up at reboot. That does NOT mean that Debian should change the default just to suit low memory devices. (Other stuff I write is for v.low memory devices which don't have swap, /tmp is on tmpfs in RAM and we just have to be careful what applications we use.) > Suggestion > == > Do not mount /tmp as tmpfs by default. Instead... No. The default is fine and sane but no default will ever satisfy every possible device. Low memory devices have many many more problems than just where /tmp is mounted. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpFujBBNhc3Q.pgp Description: PGP signature
