Re: dh_movefiles, tar vs. mv
Christoph == Christoph Berg [EMAIL PROTECTED] writes: Christoph As I understood it, the question was about moving stuff Christoph from debian/tmp to debian/package. The stuff in Christoph debian/tmp should get removed by the clean target Christoph anyway, so it doesn't hurt to move instead of copying Christoph it. A benefit of moving files, rather then copying, is that you get to see at a glance what files your package left behind and missed in debian/tmp (e.g. if upstream adds new files to the packages but doesn't document these additions). -- Brian May [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
* Brian May [Mon, 28 Feb 2005 09:05:01 +1100]: A benefit of moving files, rather then copying, is that you get to see at a glance what files your package left behind and missed in debian/tmp (e.g. if upstream adds new files to the packages but doesn't document these additions). FWIW, there is always dh_install --no-act --list-missing. -- Adeodato Simó EM: asp16 [ykwim] alu.ua.es | PK: DA6AE621 He has never been known to use a word that might send a reader to the dictionary. -- William Faulkner (about Ernest Hemingway) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
On Fri, 2005-02-25 at 11:32 -0800, Oliver Kurth wrote: On Fri, 2005-02-25 at 20:25 +0100, GOMBAS Gabor wrote: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Kster wrote: Correct. So, why not use mv? Add a new --move flag to dh_installfiles, come up with some exact numbers showing the build time/disk usage savings for your favorite Big Package (hard numbers usually very helpful for promoting new features), and send the numbers together with the patch to the debhelper maintainer. Someone already mentioned that a complex package might want to install the same file to multiple different locations, so making this the default is probably not feasible. How about hard linking with ln instead? You would have the best of both approaches: it is fast, and still possible to have the same file in multiple packages. I believe Python distutils (like autoconf/automake for Python) uses this approach for its various build/dist targets, and there don't seem to have been any problems/complaints. It also cuts down on hard drive space requirements. However, it probably shouldn't be default. A hard link would be a pretty incompatible change if someone modifies the file after it's been dh_installed (I don't have any concrete examples, but I suspect something does it, if only because 13000 packages guarantees every nasty hack appears at least once :). -- Joe Wreschnig [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: dh_movefiles, tar vs. mv
Oliver Kurth [EMAIL PROTECTED] schrieb: On Fri, 2005-02-25 at 20:25 +0100, GOMBAS Gabor wrote: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Correct. So, why not use mv? Add a new --move flag to dh_installfiles, come up with some exact numbers showing the build time/disk usage savings for your favorite Big Package (hard numbers usually very helpful for promoting new features), and send the numbers together with the patch to the debhelper maintainer. Someone already mentioned that a complex package might want to install the same file to multiple different locations, so making this the default is probably not feasible. How about hard linking with ln instead? You would have the best of both approaches: it is fast, and still possible to have the same file in multiple packages. I didn't look closely, but I think it might need quite some changes to the code. It seems dh_install uses cp -a for directories, and you cannot use hard links with directories (at least not generally, here on my ext3 $HOME it does not work. And I suspect it could end in great mess). Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
* Frank Küster [Sat, 26 Feb 2005 19:34:45 +0100]: I didn't look closely, but I think it might need quite some changes to the code. It seems dh_install uses cp -a for directories, and you cannot use hard links with directories (at least not generally, here on my ext3 $HOME it does not work. And I suspect it could end in great mess). I remember that I once modified my dh_install to use cp -al. That will make each file be a hardlink, even if you copy a dir. It's fast. I wouldn't mind that dh_install accepted an option to behave like that. -- Adeodato Simó EM: asp16 [ykwim] alu.ua.es | PK: DA6AE621 Taking any religion too seriously --even the Church of Emacs-- can be hazardous to your health. -- Richard Stallman -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
Colin Watson [EMAIL PROTECTED] wrote: My point is that half the reason why dh_install was introduced instead of dh_movefiles is to copy files, not move them, and it seems odd that now there's a thread asking how to get dh_install to move files. If you want to move files, don't use dh_install. I never asked how to get dh_install to move files. I asked what the reasons are why neither dh_install nor dh_movefiles use it. And it seems to me that the summary of the answers is: It is unwanted in most cases, in particular because of idempotency. But in cases where idempotency is less important than other things - time and disk space savings - there is no technical reason not to use it in package building. Using hard links may be an alternative, but not without major changes to the code. I have not yet decided whether I'm going to prepare a patch for dh_install, for an optional --move behavior, or just use it in a handcrafted way in my packaging script. Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
On Saturday 26 February 2005 01:37 pm, Adeodato Simó wrote: * Frank Küster [Sat, 26 Feb 2005 19:34:45 +0100]: I didn't look closely, but I think it might need quite some changes to the code. It seems dh_install uses cp -a for directories, and you cannot use hard links with directories (at least not generally, here on my ext3 $HOME it does not work. And I suspect it could end in great mess). I remember that I once modified my dh_install to use cp -al. That will make each file be a hardlink, even if you copy a dir. It's fast. I wouldn't mind that dh_install accepted an option to behave like that. See bug #296917. Daniel -- /--- Daniel Burrows [EMAIL PROTECTED] --\ |All generalizations are dangerous. | \ Evil Overlord, Inc: http://www.eviloverlord.com --/ pgpmqh7SGPS8n.pgp Description: PGP signature
Re: dh_movefiles, tar vs. mv
* Daniel Burrows [Sat, 26 Feb 2005 13:42:28 -0500]: On Saturday 26 February 2005 01:37 pm, Adeodato Simó wrote: I remember that I once modified my dh_install to use cp -al. That will make each file be a hardlink, even if you copy a dir. It's fast. I wouldn't mind that dh_install accepted an option to behave like that. See bug #296917. Ah yes, precisely. :) -- Adeodato Simó EM: asp16 [ykwim] alu.ua.es | PK: DA6AE621 So, irregular/impure/non-elegant syntax doesn't bother me. Shit, I speak English. Mainly I just want to type less. -- William Morgan, on [ruby-talk:131589] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
Eric Dorland [EMAIL PROTECTED] schrieb: * Frank Küster ([EMAIL PROTECTED]) wrote: Hi, dh_movefiles internally uses tar to move file contents. I'm not sure why it doesn't use mv, is it because mv moves the file block-by-block and thus starts removing parts of the file before it is completely written, and hence is less save? Anyway: If I am only going to move complete subdirectories from the temp tree to the package trees, is it in this case safe to use mv? It's much faster, and it would safe space (because dh_movefiles only removes the originals after the complete tarball has been extracted). Uhh, who cares? dh_movefiles has been superseded by dh_install. Well, fine. But the question remains: dh_install uses cp, not mv. What is the problem with using mv? And would it be safe to use mv if I only move complete directories? TIA, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
On Friday 25 February 2005 12:36 pm, Frank Küster wrote: Well, fine. But the question remains: dh_install uses cp, not mv. What is the problem with using mv? And would it be safe to use mv if I only move complete directories? I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. Anyway, I thought you were joking in your first message, but it looks like you're serious, so I'll answer this time. If you're copying between files on the same device, mv will use the rename(2) system call, which is an atomic operation: ie, it doesn't copy the source files at all, it just links them into the target directory. If you're copying between devices, mv will presumably copy the whole file before deleting it -- to actually remove a file block-by-block would mean a whole lot of totally pointless extra work in order to make the program less robust (there's no direct way to delete the first block of a file, so you'd have to either copy from the back or shift the whole file back a block at a time and then truncate it). Daniel -- /--- Daniel Burrows [EMAIL PROTECTED] --\ |Is it too late to extricate myself| | from this plot line? | |Yes. -- Fluble | \-- (if (not (understand-this)) (go-to http://www.schemers.org)) ---/ pgpX29TkvNl7W.pgp Description: PGP signature
Re: dh_movefiles, tar vs. mv
Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Anyway, I thought you were joking in your first message, but it looks like you're serious, so I'll answer this time. If you're copying between files on the same device, mv will use the rename(2) system call, which is an atomic operation: ie, it doesn't copy the source files at all, it just links them into the target directory. If you're copying between devices, mv will presumably copy the whole file before deleting it -- to actually remove a file block-by-block would mean a whole lot of totally pointless extra work in order to make the program less robust (there's no direct way to delete the first block of a file, so you'd have to either copy from the back or shift the whole file back a block at a time and then truncate it). I doubt that any subdirs of a package build directory will ever be on a different mount point than the directory itself ;-) Christoph -- [EMAIL PROTECTED] | http://www.df7cb.de/ signature.asc Description: Digital signature
Re: dh_movefiles, tar vs. mv
* Frank Küster ([EMAIL PROTECTED]) wrote: Eric Dorland [EMAIL PROTECTED] schrieb: * Frank Küster ([EMAIL PROTECTED]) wrote: Hi, dh_movefiles internally uses tar to move file contents. I'm not sure why it doesn't use mv, is it because mv moves the file block-by-block and thus starts removing parts of the file before it is completely written, and hence is less save? Anyway: If I am only going to move complete subdirectories from the temp tree to the package trees, is it in this case safe to use mv? It's much faster, and it would safe space (because dh_movefiles only removes the originals after the complete tarball has been extracted). Uhh, who cares? dh_movefiles has been superseded by dh_install. Well, fine. But the question remains: dh_install uses cp, not mv. What is the problem with using mv? And would it be safe to use mv if I only move complete directories? Well one reason is sometimes (in multipackage builds) you want to have the same file in 2 different packages. Also, the less side-effects during build time the better for debugging. Eg since dh_install is idempotent I can run my install target multiple times it will work. That won't work with dh_movefiles. OTOH if you have a massively big package, dh_install would be painful, especially on some of the buildds. -- Eric Dorland [EMAIL PROTECTED] ICQ: #61138586, Jabber: [EMAIL PROTECTED] 1024D/16D970C6 097C 4861 9934 27A0 8E1C 2B0A 61E9 8ECF 16D9 70C6 -BEGIN GEEK CODE BLOCK- Version: 3.12 GCS d- s++: a-- C+++ UL+++ P++ L++ E++ W++ N+ o K- w+ O? M++ V-- PS+ PE Y+ PGP++ t++ 5++ X+ R tv++ b+++ DI+ D+ G e h! r- y+ --END GEEK CODE BLOCK-- signature.asc Description: Digital signature
Re: dh_movefiles, tar vs. mv
Christoph Berg [EMAIL PROTECTED] schrieb: Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Indeed. Especially when people tell me that dh_install is the successor of dh_movefiles, which even has move in its name... Anyway, I thought you were joking in your first message, but it looks like you're serious, so I'll answer this time. If you're copying between files on the same device, mv will use the rename(2) system call, which is an atomic operation: ie, it doesn't copy the source files at all, it just links them into the target directory. That's why I want to use it. It would speed things up tremendously for packages with lots of files that need to be separated into different binary packages. If you're copying between devices, mv will presumably copy the whole file before deleting it -- to actually remove a file block-by-block would mean a whole lot of totally pointless extra work in order to make the program less robust (there's no direct way to delete the first block of a file, so you'd have to either copy from the back or shift the whole file back a block at a time and then truncate it). I doubt that any subdirs of a package build directory will ever be on a different mount point than the directory itself ;-) Correct. So, why not use mv? Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
Eric Dorland [EMAIL PROTECTED] wrote: * Frank Küster ([EMAIL PROTECTED]) wrote: Well, fine. But the question remains: dh_install uses cp, not mv. What is the problem with using mv? And would it be safe to use mv if I only move complete directories? Well one reason is sometimes (in multipackage builds) you want to have the same file in 2 different packages. Well, then I don't use mv. Note that I didn't request that dh_install use mv from now on (although an option to do this might be a nice idea). What I asked is whether there are any real reasons why I shouldn't do it. If dh_install doesn't do it this way, I can still do it manually. Also, the less side-effects during build time the better for debugging. Eg since dh_install is idempotent I can run my install target multiple times it will work. That won't work with dh_movefiles. Well, that's of course a nice thing to have. However, I think in my case its value is limited, since there is lot of other code during install time, like moving things in the installed trees, moving and symlinking, and especially removing some files completely. I tried to make it idempotent, but that seemed to be a very hard task, and I gave up. OTOH if you have a massively big package, dh_install would be painful, especially on some of the buildds. Yes, that's why I'm asking. Well, it's mainly about an architecture: all package, but its painful for *my* machine... Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
On Fri, Feb 25, 2005 at 01:14:00PM -0500, Daniel Burrows wrote: Anyway, I thought you were joking in your first message, but it looks like you're serious, so I'll answer this time. If you're copying between files on the same device, mv will use the rename(2) system call, which is an atomic operation: Well, if we are nitpicking, then rename(2) is atomic only in the sense that after it returns, you can either access the old name (if rename failed) or the new name (if it succeeded) (modulo of course journaling, disk write cache, yadda yadda if you want to extend the atomicity over a crash). _But_ it is not neccessarily atomic wrt. other operations, especially getdents(2). So it is equally possible that a readdir(3) during the rename(2) returns the old name, the new name, both or neither, depending on file system implementation and how the parent directory(/ies) is(/are) actually allocated on the disk. Not very important for dh_install, though :-) Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Kster wrote: Correct. So, why not use mv? Add a new --move flag to dh_installfiles, come up with some exact numbers showing the build time/disk usage savings for your favorite Big Package (hard numbers usually very helpful for promoting new features), and send the numbers together with the patch to the debhelper maintainer. Someone already mentioned that a complex package might want to install the same file to multiple different locations, so making this the default is probably not feasible. Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Christoph Berg [EMAIL PROTECTED] schrieb: Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Indeed. Especially when people tell me that dh_install is the successor of dh_movefiles, which even has move in its name... debhelper (4.0.0) unstable; urgency=low * dh_movefiles has long been a sore point in debhelper. Inherited from debstd, its interface and implementation suck, and I have maintained it while never really deigning to use it. Now there is a remplacment: dh_install, which ... - copies files, doesn't move them. Closes: #75360, #82649 Cheers, -- Colin Watson [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
On Fri, 2005-02-25 at 20:25 +0100, GOMBAS Gabor wrote: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Correct. So, why not use mv? Add a new --move flag to dh_installfiles, come up with some exact numbers showing the build time/disk usage savings for your favorite Big Package (hard numbers usually very helpful for promoting new features), and send the numbers together with the patch to the debhelper maintainer. Someone already mentioned that a complex package might want to install the same file to multiple different locations, so making this the default is probably not feasible. How about hard linking with ln instead? You would have the best of both approaches: it is fast, and still possible to have the same file in multiple packages. Greetings, Oliver signature.asc Description: This is a digitally signed message part
Re: dh_movefiles, tar vs. mv
Colin Watson [EMAIL PROTECTED] schrieb: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Christoph Berg [EMAIL PROTECTED] schrieb: Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Indeed. Especially when people tell me that dh_install is the successor of dh_movefiles, which even has move in its name... debhelper (4.0.0) unstable; urgency=low * dh_movefiles has long been a sore point in debhelper. Inherited from debstd, its interface and implementation suck, and I have maintained it while never really deigning to use it. Now there is a remplacment: dh_install, which ... - copies files, doesn't move them. Closes: #75360, #82649 What do you want to say with this? Do you want to tell me that using mv is bad? If yes, why? It's not in the bug reports. Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: dh_movefiles, tar vs. mv
Frank Küster wrote: Colin Watson [EMAIL PROTECTED] schrieb: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Christoph Berg [EMAIL PROTECTED] schrieb: Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Indeed. Especially when people tell me that dh_install is the successor of dh_movefiles, which even has move in its name... debhelper (4.0.0) unstable; urgency=low * dh_movefiles has long been a sore point in debhelper. Inherited from debstd, its interface and implementation suck, and I have maintained it while never really deigning to use it. Now there is a remplacment: dh_install, which ... - copies files, doesn't move them. Closes: #75360, #82649 What do you want to say with this? Do you want to tell me that using mv is bad? If yes, why? It's not in the bug reports. The changelog entry explains why dh_movefiles is implemented using tar. The bug reports explain a significant problem with using mv: loss of idempotency. -- see shy jo signature.asc Description: Digital signature
Re: dh_movefiles, tar vs. mv
On Fri, Feb 25, 2005 at 08:59:00PM +0100, Frank Küster wrote: Colin Watson [EMAIL PROTECTED] schrieb: On Fri, Feb 25, 2005 at 07:54:27PM +0100, Frank Küster wrote: Christoph Berg [EMAIL PROTECTED] schrieb: Re: Daniel Burrows in [EMAIL PROTECTED] I'd imagine that it doesn't use mv for the same reason install doesn't; ie, its purpose is to COPY files, not MOVE them. As I understood it, the question was about moving stuff from debian/tmp to debian/package. The stuff in debian/tmp should get removed by the clean target anyway, so it doesn't hurt to move instead of copying it. Indeed. Especially when people tell me that dh_install is the successor of dh_movefiles, which even has move in its name... debhelper (4.0.0) unstable; urgency=low * dh_movefiles has long been a sore point in debhelper. Inherited from debstd, its interface and implementation suck, and I have maintained it while never really deigning to use it. Now there is a remplacment: dh_install, which ... - copies files, doesn't move them. Closes: #75360, #82649 What do you want to say with this? Do you want to tell me that using mv is bad? If yes, why? It's not in the bug reports. My point is that half the reason why dh_install was introduced instead of dh_movefiles is to copy files, not move them, and it seems odd that now there's a thread asking how to get dh_install to move files. If you want to move files, don't use dh_install. Cheers, -- Colin Watson [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_movefiles, tar vs. mv
* Frank Küster ([EMAIL PROTECTED]) wrote: Hi, dh_movefiles internally uses tar to move file contents. I'm not sure why it doesn't use mv, is it because mv moves the file block-by-block and thus starts removing parts of the file before it is completely written, and hence is less save? Anyway: If I am only going to move complete subdirectories from the temp tree to the package trees, is it in this case safe to use mv? It's much faster, and it would safe space (because dh_movefiles only removes the originals after the complete tarball has been extracted). Uhh, who cares? dh_movefiles has been superseded by dh_install. -- Eric Dorland [EMAIL PROTECTED] ICQ: #61138586, Jabber: [EMAIL PROTECTED] 1024D/16D970C6 097C 4861 9934 27A0 8E1C 2B0A 61E9 8ECF 16D9 70C6 -BEGIN GEEK CODE BLOCK- Version: 3.12 GCS d- s++: a-- C+++ UL+++ P++ L++ E++ W++ N+ o K- w+ O? M++ V-- PS+ PE Y+ PGP++ t++ 5++ X+ R tv++ b+++ DI+ D+ G e h! r- y+ --END GEEK CODE BLOCK-- signature.asc Description: Digital signature
Re: dh_movefiles, tar vs. mv
Eric Dorland [EMAIL PROTECTED] wrote: Uhh, who cares? dh_movefiles has been superseded by dh_install. Oh, thank you for pointing out the obvious. Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
