Re: walking a directory with very many files

2009-07-08 Thread Lawrence D'Oliveiro
In message , Lie Ryan wrote: > Lawrence D'Oliveiro wrote: > >> ... certainly it is characteristic of GUIs to show you all 400,000 files >> in a directory, or at least try to do so, and either hang for half an >> hour or run out of memory and crash, rather than give you some >> intelligent way of p

Re: walking a directory with very many files

2009-06-24 Thread Lie Ryan
Lawrence D'Oliveiro wrote: > In message , Steven > D'Aprano wrote: > >> On Tue, 23 Jun 2009 10:29:21 -0400, Mel wrote: >> >>> Steven D'Aprano wrote: >>> Lawrence D'Oliveiro wrote: >> Ok, now pipe ls to less, take three days to browse through all the >> filenames to locate the fi

Re: walking a directory with very many files

2009-06-23 Thread Lawrence D'Oliveiro
In message , Steven D'Aprano wrote: > On Tue, 23 Jun 2009 10:29:21 -0400, Mel wrote: > >> Steven D'Aprano wrote: >> >>> Lawrence D'Oliveiro wrote: >>> > Ok, now pipe ls to less, take three days to browse through all the > filenames to locate the file you want to see. Sounds

Re: walking a directory with very many files

2009-06-23 Thread Steven D'Aprano
On Tue, 23 Jun 2009 10:29:21 -0400, Mel wrote: > Steven D'Aprano wrote: > >> Lawrence D'Oliveiro wrote: >> Ok, now pipe ls to less, take three days to browse through all the filenames to locate the file you want to see. >>> >>> Sounds like you're approaching the issue with a GUI-centr

Re: walking a directory with very many files

2009-06-23 Thread Mel
Steven D'Aprano wrote: > Lawrence D'Oliveiro wrote: > >>> Ok, now pipe ls to less, take three days to browse through all the >>> filenames to locate the file you want to see. >> >> Sounds like you're approaching the issue with a GUI-centric mentality, >> which is completely hopeless at dealing w

Re: walking a directory with very many files

2009-06-21 Thread Steven D'Aprano
Lawrence D'Oliveiro wrote: >> Ok, now pipe ls to less, take three days to browse through all the >> filenames to locate the file you want to see. > > Sounds like you're approaching the issue with a GUI-centric mentality, > which is completely hopeless at dealing with this sort of situation. Pipi

Re: walking a directory with very many files

2009-06-20 Thread Lawrence D'Oliveiro
In message , Lie Ryan wrote: > Lawrence D'Oliveiro wrote: > >> In message <%zv_l.19493$y61.5...@news-server.bigpond.net.au>, Lie Ryan >> wrote: >> >>> Yeah, it might be possible to just mv the file from outside, but not >>> being able to enter a directory just because you've got too many files >

Re: walking a directory with very many files

2009-06-20 Thread Lawrence D'Oliveiro
In message <20090619134015.349ba...@malediction>, Mike Kazantsev wrote: > On Fri, 19 Jun 2009 17:53:40 +1200 > Lawrence D'Oliveiro wrote: > >> In message <20090618081423.2e035...@coercion>, Mike Kazantsev wrote: >> >> > On Thu, 18 Jun 2009 10:33:49 +1200 >> > Lawrence D'Oliveiro wrote: >> > >

Re: walking a directory with very many files

2009-06-19 Thread Lie Ryan
Lawrence D'Oliveiro wrote: > In message <%zv_l.19493$y61.5...@news-server.bigpond.net.au>, Lie Ryan > wrote: > >> Yeah, it might be possible to just mv the file from outside, but not >> being able to enter a directory just because you've got too many files >> in it is kind of silly. > > Sounds l

Re: walking a directory with very many files

2009-06-19 Thread Mike Kazantsev
On Fri, 19 Jun 2009 17:53:40 +1200 Lawrence D'Oliveiro wrote: > In message <20090618081423.2e035...@coercion>, Mike Kazantsev wrote: > > > On Thu, 18 Jun 2009 10:33:49 +1200 > > Lawrence D'Oliveiro wrote: > > > >> In message <20090617214535.10866...@coercion>, Mike Kazantsev > >> wrote: > >>

Re: walking a directory with very many files

2009-06-18 Thread Lawrence D'Oliveiro
In message <%zv_l.19493$y61.5...@news-server.bigpond.net.au>, Lie Ryan wrote: > Yeah, it might be possible to just mv the file from outside, but not > being able to enter a directory just because you've got too many files > in it is kind of silly. Sounds like a problem with your file/directory-m

Re: walking a directory with very many files

2009-06-18 Thread Lawrence D'Oliveiro
In message , Ethan Furman wrote: > Lawrence D'Oliveiro wrote: >> In message <20090617214535.10866...@coercion>, Mike Kazantsev wrote: >> >> >>>On Wed, 17 Jun 2009 23:04:37 +1200 >>>Lawrence D'Oliveiro wrote: >>> >>> In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: >>

Re: walking a directory with very many files

2009-06-18 Thread Lawrence D'Oliveiro
In message <20090618081423.2e035...@coercion>, Mike Kazantsev wrote: > On Thu, 18 Jun 2009 10:33:49 +1200 > Lawrence D'Oliveiro wrote: > >> In message <20090617214535.10866...@coercion>, Mike Kazantsev wrote: >> >>> On Wed, 17 Jun 2009 23:04:37 +1200 >>> Lawrence D'Oliveiro wrote: >>> In

Re: walking a directory with very many files

2009-06-18 Thread Lie Ryan
Mike Kazantsev wrote: > On Wed, 17 Jun 2009 03:42:02 GMT > Lie Ryan wrote: > >> Mike Kazantsev wrote: >>> In fact, on modern filesystems it doesn't matter whether you >>> accessing /path/f9e95ea4926a4 with million files in /path >>> or /path/f/9/e/95ea with only hundred of them in each path. Form

Re: walking a directory with very many files

2009-06-17 Thread Ethan Furman
Lawrence D'Oliveiro wrote: In message <20090617214535.10866...@coercion>, Mike Kazantsev wrote: On Wed, 17 Jun 2009 23:04:37 +1200 Lawrence D'Oliveiro wrote: In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: On Wed, 17 Jun 2009 17:53:33 +1200 Lawrence D'Oliveiro w

Re: walking a directory with very many files

2009-06-17 Thread Mike Kazantsev
On Thu, 18 Jun 2009 10:33:49 +1200 Lawrence D'Oliveiro wrote: > In message <20090617214535.10866...@coercion>, Mike Kazantsev wrote: > > > On Wed, 17 Jun 2009 23:04:37 +1200 > > Lawrence D'Oliveiro wrote: > > > >> In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: > >> >

Re: walking a directory with very many files

2009-06-17 Thread Lawrence D'Oliveiro
In message <20090617214535.10866...@coercion>, Mike Kazantsev wrote: > On Wed, 17 Jun 2009 23:04:37 +1200 > Lawrence D'Oliveiro wrote: > >> In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: >> >>> On Wed, 17 Jun 2009 17:53:33 +1200 >>> Lawrence D'Oliveiro wrote: >>>

Re: walking a directory with very many files

2009-06-17 Thread Mike Kazantsev
On Wed, 17 Jun 2009 23:04:37 +1200 Lawrence D'Oliveiro wrote: > In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: > > > On Wed, 17 Jun 2009 17:53:33 +1200 > > Lawrence D'Oliveiro wrote: > > > >> > Why not use hex representation of md5/sha1-hashed id as a path, > >> > arra

Re: walking a directory with very many files

2009-06-17 Thread Scott David Daniels
Mike Kazantsev wrote: On Wed, 17 Jun 2009 14:52:28 +1200 Lawrence D'Oliveiro wrote: In message <234b19ac-7baf-4356-9fe5-37d00146d...@z9g2000yqi.googlegroups.com>, thebjorn wrote: Not proud of this, but...: [django] www4:~/datakortet/media$ ls bfpbilder|wc -l 174197 all .jpg files between

Re: walking a directory with very many files

2009-06-17 Thread Lawrence D'Oliveiro
In message <20090617142431.2b25f...@malediction>, Mike Kazantsev wrote: > On Wed, 17 Jun 2009 17:53:33 +1200 > Lawrence D'Oliveiro wrote: > >> > Why not use hex representation of md5/sha1-hashed id as a path, >> > arranging them like /path/f/9/e/95ea4926a4 ? >> > >> > That way, you won't have t

Re: walking a directory with very many files

2009-06-17 Thread Mike Kazantsev
On Wed, 17 Jun 2009 17:53:33 +1200 Lawrence D'Oliveiro wrote: > > Why not use hex representation of md5/sha1-hashed id as a path, > > arranging them like /path/f/9/e/95ea4926a4 ? > > > > That way, you won't have to deal with many-files-in-path problem ... > > Why is that a problem? So you can

Re: walking a directory with very many files

2009-06-16 Thread Lawrence D'Oliveiro
In message <20090617091858.432f8...@malediction>, Mike Kazantsev wrote: > On Wed, 17 Jun 2009 14:52:28 +1200 > Lawrence D'Oliveiro wrote: > >> In message >> <234b19ac-7baf-4356-9fe5-37d00146d...@z9g2000yqi.googlegroups.com>, >> thebjorn wrote: >> >> > Not proud of this, but...: >> > >> > [djan

Re: walking a directory with very many files

2009-06-16 Thread Lawrence D'Oliveiro
In message , Jean-Paul Calderone wrote: > The problem is that POSIX specifies the fields with types like off_t and > ino_t. Since ctypes doesn't know anything about these types, application > code has to specify their size and other attributes. As these vary from > platform to platform, you can

Re: walking a directory with very many files

2009-06-16 Thread Mike Kazantsev
On Wed, 17 Jun 2009 03:42:02 GMT Lie Ryan wrote: > Mike Kazantsev wrote: > > In fact, on modern filesystems it doesn't matter whether you > > accessing /path/f9e95ea4926a4 with million files in /path > > or /path/f/9/e/95ea with only hundred of them in each path. Former > > case (all-in-one-path)

Re: walking a directory with very many files

2009-06-16 Thread Lie Ryan
Mike Kazantsev wrote: > On Wed, 17 Jun 2009 14:52:28 +1200 > Lawrence D'Oliveiro wrote: > >> In message >> <234b19ac-7baf-4356-9fe5-37d00146d...@z9g2000yqi.googlegroups.com>, >> thebjorn wrote: >> >>> Not proud of this, but...: >>> >>> [django] www4:~/datakortet/media$ ls bfpbilder|wc -l >>> 17

Re: walking a directory with very many files

2009-06-16 Thread Mike Kazantsev
On Wed, 17 Jun 2009 14:52:28 +1200 Lawrence D'Oliveiro wrote: > In message > <234b19ac-7baf-4356-9fe5-37d00146d...@z9g2000yqi.googlegroups.com>, > thebjorn wrote: > > > Not proud of this, but...: > > > > [django] www4:~/datakortet/media$ ls bfpbilder|wc -l > > 174197 > > > > all .jpg files b

Re: walking a directory with very many files

2009-06-16 Thread Lawrence D'Oliveiro
In message <234b19ac-7baf-4356-9fe5-37d00146d...@z9g2000yqi.googlegroups.com>, thebjorn wrote: > Not proud of this, but...: > > [django] www4:~/datakortet/media$ ls bfpbilder|wc -l > 174197 > > all .jpg files between 40 and 250KB with the path stored in a database > field... *sigh* Why not p

Re: walking a directory with very many files

2009-06-14 Thread Lawrence D'Oliveiro
In message , Andre Engels wrote: > On Sun, Jun 14, 2009 at 6:35 PM, tom wrote: > >> in other languages one can avoid generating such an object by walking >> a directory as a liked list. I suppose it depends how well-liked it is. Nerdy lists may work better, but they tend not to be liked. > Wha