Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-16 Thread Neil Bothwick
On Tue, 14 Aug 2012 15:54:17 +0200, Daniel Troeder wrote: > > sys-process/incron ? > Uh... didn't know that one! ... very interesting :) > > Have you used it? Yes... > How does it perform if there are lots of modifications going on? > Does it have a throttle against fork bombing? > must-read-

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-15 Thread Bill Kenworthy
On Wed, 2012-08-15 at 15:31 +0800, Bill Kenworthy wrote: > On Tue, 2012-08-14 at 18:36 +0200, Helmut Jarausch wrote: > > On 08/14/2012 04:07:39 AM, Adam Carter wrote: > > > > I think btrfs probably is meant to provide a lot of the modern > > > > features like reiser4 or xfs > > > > > > Unfortunate

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-15 Thread Bill Kenworthy
On Tue, 2012-08-14 at 18:36 +0200, Helmut Jarausch wrote: > On 08/14/2012 04:07:39 AM, Adam Carter wrote: > > > I think btrfs probably is meant to provide a lot of the modern > > > features like reiser4 or xfs > > > > Unfortunately btrfs is still generally slower than ext4 for example. > > Checkou

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Alecks Gates
On Tue, Aug 14, 2012 at 3:17 PM, Michael Mol wrote: > On Tue, Aug 14, 2012 at 3:55 PM, Alecks Gates wrote: >> >> On Tue, Aug 14, 2012 at 12:50 PM, Michael Hampicke >> wrote: >> > Am 14.08.2012 19:42, schrieb Volker Armin Hemmann: >> >> Am Dienstag, 14. August 2012, 13:21:35 schrieb Jason Weisber

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Michael Mol
On Tue, Aug 14, 2012 at 3:55 PM, Alecks Gates wrote: > On Tue, Aug 14, 2012 at 12:50 PM, Michael Hampicke > wrote: > > Am 14.08.2012 19:42, schrieb Volker Armin Hemmann: > >> Am Dienstag, 14. August 2012, 13:21:35 schrieb Jason Weisberger: > >>> Sure, but wouldn't compression make write operatio

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Alecks Gates
On Tue, Aug 14, 2012 at 12:50 PM, Michael Hampicke wrote: > Am 14.08.2012 19:42, schrieb Volker Armin Hemmann: >> Am Dienstag, 14. August 2012, 13:21:35 schrieb Jason Weisberger: >>> Sure, but wouldn't compression make write operations slower? And isn't he >>> looking for performance? >> >> not r

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Paul Hartman
On Tue, Aug 14, 2012 at 12:05 PM, Pandu Poluan wrote: > > On Aug 14, 2012 11:42 PM, "Helmut Jarausch" > wrote: >> >> On 08/14/2012 04:07:39 AM, Adam Carter wrote: >>> >>> > I think btrfs probably is meant to provide a lot of the modern >>> > features like reiser4 or xfs >>> >>> Unfortunately btrf

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Michael Hampicke
Am 14.08.2012 19:42, schrieb Volker Armin Hemmann: > Am Dienstag, 14. August 2012, 13:21:35 schrieb Jason Weisberger: >> Sure, but wouldn't compression make write operations slower? And isn't he >> looking for performance? > > not really. As long as the CPU can compress faster than the disk can w

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Michael Hampicke
Am 14.08.2012 19:21, schrieb Jason Weisberger: > Sure, but wouldn't compression make write operations slower? And isn't he > looking for performance? > On Aug 14, 2012 1:14 PM, "Pandu Poluan" wrote: > >> >> On Aug 14, 2012 11:42 PM, "Helmut Jarausch" >> wrote: >>> >>> On 08/14/2012 04:07:39 AM,

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Michael Hampicke
Am 14.08.2012 10:21, schrieb Daniel Troeder: > On 13.08.2012 16:53, Michael Hampicke wrote: >> 2012/8/13 Daniel Troeder > 3rd thought: purging old files with "find"? your cache system should >> have some kind of DB that holds that information. >> 3: Well, it's a 3rd party application that -

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Volker Armin Hemmann
Am Mittwoch, 15. August 2012, 00:05:40 schrieb Pandu Poluan: > > Are the support tools for btrfs (fsck, defrag, etc.) already complete? no -- #163933

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Michael Hampicke
Am 14.08.2012 16:00, schrieb Florian Philipp: > Am 13.08.2012 20:18, schrieb Michael Hampicke: >> Am 13.08.2012 19:14, schrieb Florian Philipp: >>> Am 13.08.2012 16:52, schrieb Michael Mol: On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke mailto:mgehampi...@gmail.com>> wrote:

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Volker Armin Hemmann
Am Dienstag, 14. August 2012, 13:21:35 schrieb Jason Weisberger: > Sure, but wouldn't compression make write operations slower? And isn't he > looking for performance? not really. As long as the CPU can compress faster than the disk can write stuff. More interessting: is btrfs trying to be smar

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Jason Weisberger
Sure, but wouldn't compression make write operations slower? And isn't he looking for performance? On Aug 14, 2012 1:14 PM, "Pandu Poluan" wrote: > > On Aug 14, 2012 11:42 PM, "Helmut Jarausch" > wrote: > > > > On 08/14/2012 04:07:39 AM, Adam Carter wrote: > >> > >> > I think btrfs probably is

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Pandu Poluan
On Aug 14, 2012 11:42 PM, "Helmut Jarausch" wrote: > > On 08/14/2012 04:07:39 AM, Adam Carter wrote: >> >> > I think btrfs probably is meant to provide a lot of the modern >> > features like reiser4 or xfs >> >> Unfortunately btrfs is still generally slower than ext4 for example. >> Checkout http:

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Helmut Jarausch
On 08/14/2012 04:07:39 AM, Adam Carter wrote: > I think btrfs probably is meant to provide a lot of the modern > features like reiser4 or xfs Unfortunately btrfs is still generally slower than ext4 for example. Checkout http://openbenchmarking.org/, eg http://openbenchmarking.org/s/ext4%20btrfs

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Florian Philipp
Am 14.08.2012 17:09, schrieb Florian Philipp: > > Retrieving files created 30+ days ago: > awk -v newest=$(date -d -5seconds +%s) ' > $1>newest{ nextfile } > { print $3 }' > s/-5seconds/-30days/ signature.asc Description: OpenPGP digital signature

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Florian Philipp
Am 14.08.2012 15:54, schrieb Daniel Troeder: > On 14.08.2012 11:46, Neil Bothwick wrote: >> On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote: >> >>> There is also the possibility to write a really small daemon (less than >>> 50 lines of C) that registers with inotify for the entire fs and >

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Florian Philipp
Am 13.08.2012 20:18, schrieb Michael Hampicke: > Am 13.08.2012 19:14, schrieb Florian Philipp: >> Am 13.08.2012 16:52, schrieb Michael Mol: >>> On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke >>> mailto:mgehampi...@gmail.com>> wrote: >>> >>> Have you indexed your ext4 partition? >>> >>>

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Daniel Troeder
On 14.08.2012 11:46, Neil Bothwick wrote: > On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote: > >> There is also the possibility to write a really small daemon (less than >> 50 lines of C) that registers with inotify for the entire fs and >> journals the file activity to a sqlite-db. > >

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Florian Philipp
Am 14.08.2012 11:46, schrieb Neil Bothwick: > On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote: > >> There is also the possibility to write a really small daemon (less than >> 50 lines of C) that registers with inotify for the entire fs and >> journals the file activity to a sqlite-db. >

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Neil Bothwick
On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote: > There is also the possibility to write a really small daemon (less than > 50 lines of C) that registers with inotify for the entire fs and > journals the file activity to a sqlite-db. sys-process/incron ? -- Neil Bothwick A friend of

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-14 Thread Daniel Troeder
On 13.08.2012 16:53, Michael Hampicke wrote: > 2012/8/13 Daniel Troeder 3rd thought: purging old files with "find"? your cache system should > have some kind of DB that holds that information. > 3: Well, it's a 3rd party application that - in theory - should take > care of removing old fil

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Adam Carter
> I think btrfs probably is meant to provide a lot of the modern > features like reiser4 or xfs Unfortunately btrfs is still generally slower than ext4 for example. Checkout http://openbenchmarking.org/, eg http://openbenchmarking.org/s/ext4%20btrfs The OS will use any spare RAM for disk caching,

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Volker Armin Hemmann
Am Montag, 13. August 2012, 15:13:03 schrieb Paul Hartman: > On Mon, Aug 13, 2012 at 8:16 AM, Michael Hampicke wrote: > > Howdy gentooers, > > > > I am looking for a filesystem that perfomes well for a cache directory. > > Here's some data on that dir: > > - cache for prescaled images files + me

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Paul Hartman
On Mon, Aug 13, 2012 at 8:16 AM, Michael Hampicke wrote: > Howdy gentooers, > > I am looking for a filesystem that perfomes well for a cache directory. > Here's some data on that dir: > - cache for prescaled images files + metadata files > - nested directory structure ( 20/2022/202231/*files* ) >

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
Am 13.08.2012 19:14, schrieb Florian Philipp: > Am 13.08.2012 16:52, schrieb Michael Mol: >> On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke >> mailto:mgehampi...@gmail.com>> wrote: >> >> Have you indexed your ext4 partition? >> >> # tune2fs -O dir_index /dev/your_partition >>

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Florian Philipp
Am 13.08.2012 16:52, schrieb Michael Mol: > On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke > mailto:mgehampi...@gmail.com>> wrote: > > Have you indexed your ext4 partition? > > # tune2fs -O dir_index /dev/your_partition > # e2fsck -D /dev/your_partition > > Hi, th

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Mol
On Mon, Aug 13, 2012 at 11:26 AM, Michael Hampicke wrote: > Am 13.08.2012 16:52, schrieb Michael Mol: > > On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke < > mgehampi...@gmail.com>wrote: > > > >> Have you indexed your ext4 partition? > >>> > >>> # tune2fs -O dir_index /dev/your_partition > >>>

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
Am 13.08.2012 16:52, schrieb Michael Mol: > On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke > wrote: > >> Have you indexed your ext4 partition? >>> >>> # tune2fs -O dir_index /dev/your_partition >>> # e2fsck -D /dev/your_partition >>> >> Hi, the dir_index is active. I guess that's why delete o

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Nilesh Govindrajan
On Mon 13 Aug 2012 08:28:15 PM IST, Michael Hampicke wrote: I guess traversing through directories may be faster with XFS, but in my experience ext4 perfoms better than XFS in regard to operations (cp, rm) on small files. I read that there are some tuning options for XFS and small

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
> > I guess traversing through directories may be faster with XFS, but in my > experience ext4 perfoms better than XFS in regard to operations (cp, rm) on > small files. > I read that there are some tuning options for XFS and small files, but > never tried it. > > But if somone seconds XFS I will

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
2012/8/13 Daniel Troeder > On 13.08.2012 15:16, Michael Hampicke wrote: > > - about 20GB > > - 100.000 directories > > - about 2 million files > > > > The system has 2x Intel Xon Quad-cores (Nehalem), 16GB of RAM and two > > 10.000rpm hard drives running a RAID1. > 1st thought: switch to SSDs > 2

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Mol
On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke wrote: > Have you indexed your ext4 partition? >> >> # tune2fs -O dir_index /dev/your_partition >> # e2fsck -D /dev/your_partition >> > Hi, the dir_index is active. I guess that's why delete operations take as > long as they take (index has to be

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
> > Have you indexed your ext4 partition? > > # tune2fs -O dir_index /dev/your_partition > # e2fsck -D /dev/your_partition > Hi, the dir_index is active. I guess that's why delete operations take as long as they take (index has to be updated every time)

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Dale
Michael Hampicke wrote: > > You should have a look at xfs. > > I used to use ext4 earlier, traversing through /usr/portage used > to be very slow. When I switched xfs, speed increased drastically. > > This might be kind of unrelated, but makes sense. > > > I guess traversing through

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Daniel Troeder
On 13.08.2012 15:16, Michael Hampicke wrote: > - about 20GB > - 100.000 directories > - about 2 million files > > The system has 2x Intel Xon Quad-cores (Nehalem), 16GB of RAM and two > 10.000rpm hard drives running a RAID1. 1st thought: switch to SSDs 2nd thought: maybe lots of writes? -> get a S

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Pandu Poluan
On Aug 13, 2012 9:01 PM, "Michael Hampicke" wrote: >> >> You should have a look at xfs. >> >> I used to use ext4 earlier, traversing through /usr/portage used to be very slow. When I switched xfs, speed increased drastically. >> >> This might be kind of unrelated, but makes sense. > > > I guess tr

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Michael Hampicke
> > You should have a look at xfs. > > I used to use ext4 earlier, traversing through /usr/portage used to be > very slow. When I switched xfs, speed increased drastically. > > This might be kind of unrelated, but makes sense. I guess traversing through directories may be faster with XFS, but in

Re: [gentoo-user] Fast file system for cache directory with lot's of files

2012-08-13 Thread Nilesh Govindrajan
On Mon 13 Aug 2012 06:46:53 PM IST, Michael Hampicke wrote: Howdy gentooers, I am looking for a filesystem that perfomes well for a cache directory. Here's some data on that dir: - cache for prescaled images files + metadata files - nested directory structure ( 20/2022/202231/*files* ) - about 2