Re: [CentOS] Fastest way of removing very large number of files?
In article <20130823104855.ga15...@mercury.spuddy.org>, Stephen Harris wrote: > On Fri, Aug 23, 2013 at 12:40:51PM +0200, Dennis Jacobfeuerborn wrote: > > I doubt saving functions calls is going to gain you anything in this > > case as 99.9% of the time the rm takes is on disk I/O. If you want to > > reduce the rm time you have to find a way to reduce the disk I/O it > > requires. > > Correct. > > If it's a whole directory (tree) that needs removing then I find > > mv dir dir.o ; mkdir dir ; chown ##:## dir; chmod ### dir ; rm -r dir.o & > > type stuff works just fine; the rm can chunk along in the background > while there's now a nice clean empty directory for the application. You can even dispense with the ## variable parts. And I would get the owner and modes correct before making the new directory appear: mkdir dir.n && chown --reference=dir dir.n && chmod --reference=dir dir.n && \ mv dir dir.o && mv dir.n dir && echo rm -fr dir.o \; echo dir.o deleted | batch That will do the removal in a batch job without hanging on to the tty, and will email you a quick note (the output from batch) when it's done. Cheers Tony -- Tony Mountifield Work: t...@softins.co.uk - http://www.softins.co.uk Play: t...@mountifield.org - http://tony.mountifield.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fastest way of removing very large number of files?
On Fri, Aug 23, 2013 at 12:40:51PM +0200, Dennis Jacobfeuerborn wrote: > I doubt saving functions calls is going to gain you anything in this > case as 99.9% of the time the rm takes is on disk I/O. If you want to > reduce the rm time you have to find a way to reduce the disk I/O it > requires. Correct. If it's a whole directory (tree) that needs removing then I find mv dir dir.o ; mkdir dir ; chown ##:## dir; chmod ### dir ; rm -r dir.o & type stuff works just fine; the rm can chunk along in the background while there's now a nice clean empty directory for the application. -- rgds Stephen ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fastest way of removing very large number of files?
On 23.08.2013 07:52, Xinyun Zhou wrote: > On Thu, 2013-08-22 at 23:40 -0600, Frank Cox wrote: >> On Fri, 23 Aug 2013 07:34:50 +0200 >> Rafał Radecki wrote: >> >>> Can you recommend something? I use an ext4 filesystem. >> >> Without actually trying to to see, I suspect that a C program that calls >> remove() would probably be faster than just about anything else. >> > > I agree with what you are saying, and I think you may even want to use > unlink() if they are all files, no dirs. remove() actually will call > unlink() so you may be able to save few function calls and logical > statements. I doubt saving functions calls is going to gain you anything in this case as 99.9% of the time the rm takes is on disk I/O. If you want to reduce the rm time you have to find a way to reduce the disk I/O it requires. Regards, Dennis ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fastest way of removing very large number of files?
- Original Message - | Hi All. | | I currently have a problem on my backup server with very large number | of | small files in a large number of directories. I would like to delete | them | as fast as possible. Currently I use: | | rsync -a --delete /empty_directory/ dir_to_clean/ | | I've read that rsync will be faster than rm or find. | Can you recommend something? I use an ext4 filesystem. | | Best regards, | Rafal. If it's on a different file system than mkfs is the fastest. ;) -- James A. Peltier Manager, IT Services - Research Computing Group Simon Fraser University - Burnaby Campus Phone : 778-782-6573 Fax : 778-782-3045 E-Mail : jpelt...@sfu.ca Website : http://www.sfu.ca/itservices “A successful person is one who can lay a solid foundation from the bricks others have thrown at them.” -David Brinkley via Luke Shaw ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fastest way of removing very large number of files?
On Thu, 2013-08-22 at 23:40 -0600, Frank Cox wrote: > On Fri, 23 Aug 2013 07:34:50 +0200 > Rafał Radecki wrote: > > > Can you recommend something? I use an ext4 filesystem. > > Without actually trying to to see, I suspect that a C program that calls > remove() would probably be faster than just about anything else. > I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements. -- Regards, Xinyun ZHOU ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Fastest way of removing very large number of files?
On Fri, 23 Aug 2013 07:34:50 +0200 Rafał Radecki wrote: > Can you recommend something? I use an ext4 filesystem. Without actually trying to to see, I suspect that a C program that calls remove() would probably be faster than just about anything else. -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Fastest way of removing very large number of files?
Hi All. I currently have a problem on my backup server with very large number of small files in a large number of directories. I would like to delete them as fast as possible. Currently I use: rsync -a --delete /empty_directory/ dir_to_clean/ I've read that rsync will be faster than rm or find. Can you recommend something? I use an ext4 filesystem. Best regards, Rafal. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
