Re: swap files
Date: Mon, 11 Oct 1999 23:04:44 -0700 From: Jay Cox [EMAIL PROTECTED] Marc Lehmann wrote: A normal user has nothing to do with the swap file, except that he will find that his disk is full and many hours later he might even find the reason It will be much more difficult for the user to find the cause if we unlink the file as soon as we create it. The normal programs used to find wasted space like "du" and "find" will not be able to locate the unlinked gimp swap file. This would be a bad thing, particularly on multiuser systems. Or (again, particularly on a multiuser system) the system administrator. Because it works. Doing it in the signal handler doesn't. If the gimp doesn't crash there is no problem. If it does crash the signal handler often isn't called at all. I can't remember the last time gimp crashed on me without the signal handler being called. I have had it hit infinite loops though... The most common infinite loop I've seen, in fact, is that when I run the Gimp in the background (I often run it as (gimp) so it doesn't become a background task I might accidentally kill) and it hits the signal handler, it goes into an infinite loop asking me to tell it what to do (continue, dump core, debug, or whatever). It shouldn't try more than a couple of times, and if it can't get any input from the user, die appropriately. In some cases, you do not have the choice. On several machines that I use at work, I have to put the swap file on NFS because all writable partitions are mounted over NFS (except for /tmp but it is too small). unlink also works over nfs. I think someone else posted in this thread a way in which unlink could actually get rid of the file before the file is closed if the file resides on an NFS volume. What usually happens is that on the server there's something spawned by cron that periodically goes around and looks for files named .NFS* which it nukes if they're too old (usually measured in days). It's possible that other NFS servers do things differently. Clients that do anything like this are completely broken. In any event, a kernel crash on the client will result in these files not being removed. The handling of removed-but-still-open files is not one of NFS's strong points. -- Robert Krawitz [EMAIL PROTECTED] http://www.tiac.net/users/rlk/ Tall Clubs International -- http://www.tall.org/ or 1-888-IM-TALL-2 Member of the League for Programming Freedom -- mail [EMAIL PROTECTED] "Linux doesn't dictate how I work, I dictate how Linux works." --Eric Crampton
Re: swap files
The following is not a real solution, but it I think it would make the user understand the necessary internals better: Add a second dialog to the "User Installation" process that explains the user Gimp's way of handling memory and allows him to choose the size of the tile-cache-size and the location of the swap-file. At this point the dialog should try to propose some reasonably alternatives: Choices for the directory should be '~/.gimp/tmp' or the the result of g_get_tmpdir(). And of course, the dialog should not give any default values and will not go away until the user makes a decision. To allow system-administrators to make this choices for their users, we skip the dialog if the global gimprc already contains entries for these settings. What do you think?? Salut, Sven
Re: swap files
Here's a better solution. I think it should work on all Unixes When gimp starts, scan the home directory for gimpswap files, and check if they are still current by trying to get an exclusive lock with an open() call. Any files that you can get a lock for, you unlink(). Finally, you create the new gimpswap with an exclusive lock. The only problem I can think of is if extensions have to open the file themselves, which I think is not the case since gimp should have some form of memory allocations from the swapfile. Cheers Bruce -Original Message- From: Sven Neumann [EMAIL PROTECTED] To: [EMAIL PROTECTED] [EMAIL PROTECTED] Date: 12 October 1999 05:34 Subject: Re: swap files The following is not a real solution, but it I think it would make the user understand the necessary internals better: Add a second dialog to the "User Installation" process that explains the user Gimp's way of handling memory and allows him to choose the size of the tile-cache-size and the location of the swap-file. At this point the dialog should try to propose some reasonably alternatives: Choices for the directory should be '~/.gimp/tmp' or the the result of g_get_tmpdir(). And of course, the dialog should not give any default values and will not go away until the user makes a decision. To allow system-administrators to make this choices for their users, we skip the dialog if the global gimprc already contains entries for these settings. What do you think?? Salut, Sven
Re: swap files
On Mon, Oct 11, 1999 at 11:04:44PM -0700, Jay Cox [EMAIL PROTECTED] wrote: A normal user has nothing to do with the swap file, except that he will find that his disk is full and many hours later he might even find the reason It will be much more difficult for the user to find the cause if we unlink the file as soon as we create it. The normal programs used to find wasted space like "du" and "find" will not be able to locate the unlinked gimp swap file. This would be a bad thing, particularly on multiuser systems. Under what circumstances will this happen? As soon as the user exits or kills a runaway gimp the soace will be gone, so no space will be wasted. I can't remember the last time gimp crashed on me without the signal handler being called. I have had it hit infinite loops though... double segfaults are not rare. I think the issue really boils down to which we think is worse: Completely invisible, but guaranteed temporary files, or visible files which (hopefully only rarely) get left around when there is a crash. Well, I don't really see a reason why gimp has to manage its temporary space unlike ANY other programs do - iso c explicity supports this kind of temporary file management. I really don't get it why we risk leaving around these files (on unix)? unlink also works over nfs. I think someone else posted in this thread a way in which unlink could actually get rid of the file before the file is closed if the file resides on an NFS volume. I did. But its quite rare and I would be surpriused if anybody could produce this case. I don't think we catch all of the possible signals curently, so we should be able to make it able to catch more crashes than it does now. Why not just fix the problem at its root? (tmpfile or unlink is the only reliable solution so far) It is not portable to assume that TMPDIR will control the location of a file created by tmpfile, so I would count it out as a solution. We can roll our own tmpfile easily. I think that would be more portable than trying to catch each and every signal. the file it would be extremly difficult for even an experienced operator to figure out why her disk was full. An inexperienced operator would probably just assume it was an OS bug and reboot the machine. Well, bugs are often difficult to diagnose. If we add the unlink call to the signal handlers and there is still a problem with swap files being left around, we could fairly easily (nearly) guarantee that any previous swapfiles left around by gimp running on the same host under the same user are deleted the next time gimp is run. (embed the user name, IP address, and process number at the top of the swap file) Thats something I could go with -- my main concern are the 300mb swap files I have to delete every month. uniquness. (mkstemp is a BSDism, is it portable?) Portable to BSD, yes ;) Another thing a tmpfile replacement (if necessary) would solve. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: swap files
On Sun, 10 Oct 1999, Federico Mena Quintero [EMAIL PROTECTED] wrote: [quoting Jay Cox] If we unlink the swap file after opening it then we have no way of knowing how much space gimp is using for it's swap file. lseek() returns the offset to which you seeked, so lseek (swap_fd, 0, SEEK_END) will give you its size. This is fine from within the Gimp. But most of the time I want to monitor the size of the swap file from a shell window, while the Gimp is performing some memory- and CPU-intensive tasks on a large image. If we unlink the file after opening it, then the file may not be visible in a directory listing. I suppose that this depends on the OS and filesystem used. On Mon, 11 Oct 1999, Marc Lehmann [EMAIL PROTECTED] wrote: a) unlink might not succeed on non-unix-systems. cure: ignore the error from unlink and try to unlink it again after closing the file This means in any case, we have to add some code that deletes the file when a signal handler is called (close and unlink in case of a crash). So if we have to do this in a signal handler anyway, why would we also do it just after opening the file? b) if the swap file is on nfs (extremely bad idea anyway) it might go away (physically) on the server when gimp does not access it for extended periods (10 minutes). In some cases, you do not have the choice. On several machines that I use at work, I have to put the swap file on NFS because all writable partitions are mounted over NFS (except for /tmp but it is too small). However, any other solution (doing it in the signal handler) is extremely annyoing (leaves too many swap files around). Are you sure about that? I think that it is more likely that the Gimp crashes and calls the signal handler than to have a power failure (which has more serious effects than just leaving a swap file around). In any case, why not use tmpfile or a similar function to create it? that function will do exactly what is required and will work on all systems (as good as it can). The problem with tmpfile() is that it is not as easy for the user to predict or change the location of the swap file. This function calls tmpnam() which relies on TMPDIR for the location of the file. But in some cases, the user might want to store the huge Gimp swap file on a (slow but large) NFS-mounted partition instead of a (fast but small) local disk, while all other applications would keep their reasonably small temporary files on a local /tmp disk. It could still be possible to use tmpfile() as long as we clear or override TMPDIR from inside the Gimp, but this is not a very elegant solution. I would prefer to have the file closed and unlinked in a signal handler. This is not a perfect solution (e.g. in case of power failure), but it has more advantages than the other solutions (or at least less disadvantages). -Raphael
Re: swap files
I think this will just give you the LENGTH of the file? So, I don't know if it matters, but ISTR that Gimp uses files with holes, and therefore LENGTH != SIZE. Oh, I didn't know this. In any case, the program *does* know (or should) know the number of tiles that it can have (tile cache size), the number of tiles in memory (referenced tiles plus unswapped tiles), and number of tiles on disk (swapped tiles). So you really don't care much about the size of the swapfile, since it can only get so big. Federico
Re: swap files
On Mon, 11 Oct 1999, Marc Lehmann wrote: In any case, why not use tmpfile or a similar function to create it? that function will do exactly what is required and will work on all systems (as good as it can). Under Linux at least, tmpfile() simply does an unlink() after opening the file. :-) The problem with tmpfile(), as it has been pointed out, is that it does not allow for changing the path to swap dir. --- David Monniaux Tel: +33 1 44 32 20 66Fax: +33 1 44 32 20 80 Laboratoire d'informatique de l'École Normale Supérieure, 45 rue d'Ulm - 75230 PARIS cedex 5 - FRANCE
Re: swap files
On Sat, Oct 09, 1999 at 07:07:59PM -0700, Jay Cox wrote: David Monniaux wrote: Would it be possible to have Gimp unlink() the files after opening them? If we unlink the swap file after opening it then we have no way of knowing how much space gimp is using for it's swap file. I don't understand: can't you fstat your file handlers to compute used space ? I think unlinking the swap files in the signal handler that is called when we crash would solve this problem. The cases where this will not work should be rare enough not to worry about. Well, power failures or cleaning people unplugging computers are not that rare... Éric Brunet
Re: swap files
On Sat, 09 Oct 1999, Adrian Likins wrote: If I recall correctly, this has been suggested a few times and made it into released versions of gimp at least once, only to be removed later for reasons I do not recall. Seems like some sort of portability issues perhaps, but its been a long time. The easiest way to remember why it got removed is probabaly to put it back in though and see who yells ;- How about putting the unlink() call inside the signal handler that traps the crashes? This behavior would be nicer than unlinking the swap file after it's creation. Also because I don't think that this will work well when gimp is running on HP-UX and NFS (but I'm not completly sure of that). -- //\/\ Marco (LM) Lamberto e-mail:[EMAIL PROTECTED] (remove 'nOsPaMz-') The Sunny Spot - http://www.geocities.com/Tokyo/1474/
Re: swap files
David Monniaux wrote: Hi all, Many people around here complain that Gimp leaves enormous swap files when crashing. It is especially a pain since they are in a "hidden" subdirectory. Would it be possible to have Gimp unlink() the files after opening them? I explain: if we use swap files as follow: open them at the beginning and never close them until the end of execution of Gimp, then if we unlink() them after opening them they will get erased when Gimp exits, whether it exists gracefully or not. I don't know how swapping works in Gimp, so feel free to point me out the obvious reason why it doesn't work. :-) [There's another method: use a wrapper script that erases the files afterwards. That's workable, but not foolproof: if somebody launches two Gimp sessions simultaneously, that'll lead to mysterious problems. Also, yes I know swap files should go to a local filesystem, but even if we do that then we'd end up with tons of swap files in /var/tmp.] If we unlink the swap file after opening it then we have no way of knowing how much space gimp is using for it's swap file. I think unlinking the swap files in the signal handler that is called when we crash would solve this problem. The cases where this will not work should be rare enough not to worry about. Jay Cox [EMAIL PROTECTED]
