Re: UFS2 limits
On Sun, 9 Nov 2008 13:10:46 Jeremy Chadwick wrote: On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote: Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? As far as I know, there is no such limit on the number of files/dirs inside of a directory. I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. If this is something you've written or have control over or can work with engineers in regards to, I recommend you change your directory naming scheme to have separate subdirectories with the first 2 or 3 letters of the directory you wish to create. E.g.: /some/place/00/00ilikezeros/* /some/place/01/01binaryheaven/* /some/place/aa/aardvarks/* /some/place/ab/abuse/* /some/place/ac/actuary/* ... /some/place/xy/xylophones/* You get the point. Traversing this structure is much more efficient, and requires very little code change on your part. Those who run nameservers that host many zones, for example, use this structure to ensure the daemon doesn't take 32498231 years to start up. What about ZFS? At some point I'll have to re-arrange things so that I have a deeper directory structure, just wondering when I'll hit the limit so I can plan in advance :-) What baffles me is why you're looking at this problem from a how can the filesystem solve this engineering mistake I made for me standpoint, rather than how can I solve this engineering mistake I made so that it doesn't impact the filesystem. Very strange. Sometimes looking at things in a different light makes all the difference. Hope this helps. P.S. -- I hope this mail makes it to you, because your From line is [EMAIL PROTECTED] (I'll be surprised if your account name really is that!). Thanks for that Jeremy. I didn't invent this structure, but I daresay I can either modify it or get the original writer to do that. I never really gave it a thought before now - it was the system I was given to work with and it's worked fine so far, except when I try to list the contents of the directory - that takes ages! All the folders are 7 digit numbers and we are up to approx. 0010500 entries (ie subdirs) so far. I guess it will just be a matter of experimenting to find the optimum number of sub-sub-directories per sub-directory :-/ Oh, and yes, that email address does work - I use it for mailing lists and other stuff where I'm likely to get spammed - it's ironic really :-) Cheers, -- Ian gpg key: http://home.swiftdsl.com.au/~imoore/no-spam.asc signature.asc Description: This is a digitally signed message part.
Re: UFS2 limits
The number of files and sub-directories is limited by the number of available inodes which is fixed at the time you create the file system (by -i argument to newfs(8)). Anyway, stick with Jeremy's advise if you do not like trouble. Jeremy Chadwick wrote: On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote: Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? As far as I know, there is no such limit on the number of files/dirs inside of a directory. I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. If this is something you've written or have control over or can work with engineers in regards to, I recommend you change your directory naming scheme to have separate subdirectories with the first 2 or 3 letters of the directory you wish to create. E.g.: /some/place/00/00ilikezeros/* /some/place/01/01binaryheaven/* /some/place/aa/aardvarks/* /some/place/ab/abuse/* /some/place/ac/actuary/* ... /some/place/xy/xylophones/* You get the point. Traversing this structure is much more efficient, and requires very little code change on your part. Those who run nameservers that host many zones, for example, use this structure to ensure the daemon doesn't take 32498231 years to start up. What about ZFS? At some point I'll have to re-arrange things so that I have a deeper directory structure, just wondering when I'll hit the limit so I can plan in advance :-) What baffles me is why you're looking at this problem from a how can the filesystem solve this engineering mistake I made for me standpoint, rather than how can I solve this engineering mistake I made so that it doesn't impact the filesystem. Very strange. Sometimes looking at things in a different light makes all the difference. Hope this helps. P.S. -- I hope this mail makes it to you, because your From line is [EMAIL PROTECTED] (I'll be surprised if your account name really is that!). -- Best regards. Hooman Fazaeli [3][EMAIL PROTECTED] Sepehr S. T. Co. Ltd. Web: [4]http://www.sepehrs.com Tel: (9821)88975701-2 Fax: (9821)88983352 References 1. mailto:[EMAIL PROTECTED] 2. mailto:[EMAIL PROTECTED] 3. mailto:[EMAIL PROTECTED] 4. http://www.sepehrs.com/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sat, Nov 08, 2008 at 06:40:46PM -0800, Jeremy Chadwick wrote:
On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote:
Hi,
I have a FreeBSD server that has about 10,500 subdirectories within a single
directory.
This number will keep rising and I assume UFS2 has a limit to the number of
sub-directories in a single directory - can anyone tell me what it is?
As far as I know, there is no such limit on the number of files/dirs
inside of a directory.
Actually there is. Each i-node on the disk contains a field telling how
many hard-links point to that inode. This field is a (signed) 16-bit value,
meaning the maximum number of hardlinks allowed is 32767.
Each subdirectory created contains a hardlink ('..') to its parent, thus
limiting the number of subdirectories to a single directory to less than 32767.
Note that this does not limit the number of files you can have in a single
directory, since normal files do not contain hardlinks to the parent
directory, but there are of course limits to the total number of files and
directories you can have on a single filesystem based on how many inodes
were created when the filesystem was first created.
--
Insert your favourite quote here.
Erik Trulsson
[EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 10:35:21AM +0100, Erik Trulsson wrote:
On Sat, Nov 08, 2008 at 06:40:46PM -0800, Jeremy Chadwick wrote:
On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote:
Hi,
I have a FreeBSD server that has about 10,500 subdirectories within a
single
directory.
This number will keep rising and I assume UFS2 has a limit to the number
of
sub-directories in a single directory - can anyone tell me what it is?
As far as I know, there is no such limit on the number of files/dirs
inside of a directory.
Actually there is. Each i-node on the disk contains a field telling how
many hard-links point to that inode. This field is a (signed) 16-bit value,
meaning the maximum number of hardlinks allowed is 32767.
Each subdirectory created contains a hardlink ('..') to its parent, thus
limiting the number of subdirectories to a single directory to less than
32767.
I hadn't even thought of that (the inode possibility did cross my mind,
but I figured inode counts would a) apply to the filesystem as a whole,
and b) apply to both files and directories.
Wow, thanks for the tip! Learning something new every day... :-)
Note that this does not limit the number of files you can have in a single
directory, since normal files do not contain hardlinks to the parent
directory, but there are of course limits to the total number of files and
directories you can have on a single filesystem based on how many inodes
were created when the filesystem was first created.
--
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 10:47:11AM +0100, Polytropon wrote: On Sun, 9 Nov 2008 10:35:21 +0100, Erik Trulsson [EMAIL PROTECTED] wrote: Note that this does not limit the number of files you can have in a single directory, since normal files do not contain hardlinks to the parent directory, but there are of course limits to the total number of files and directories you can have on a single filesystem based on how many inodes were created when the filesystem was first created. Maybe this sounds stupid, but... given that a file system can hold n entries. What happens when a program tries to create file number n + 1? The call to the open(2) system call will fail for one of the reasons given in the manual page. I do ask this in order to explore if this could have been the reason for my massive data loss and UFS file system corruption. My first point of inquiry in such a case would be if the hardware is OK. If you're using (P|S)ATA|SCSI-3 devices, install smartmontools from ports and test the disk with smartctl(8). Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgp0o7tO9DuGA.pgp Description: PGP signature
Re: UFS2 limits
On Sun, 9 Nov 2008 10:35:21 +0100, Erik Trulsson [EMAIL PROTECTED] wrote: Note that this does not limit the number of files you can have in a single directory, since normal files do not contain hardlinks to the parent directory, but there are of course limits to the total number of files and directories you can have on a single filesystem based on how many inodes were created when the filesystem was first created. Maybe this sounds stupid, but... given that a file system can hold n entries. What happens when a program tries to create file number n + 1? I do ask this in order to explore if this could have been the reason for my massive data loss and UFS file system corruption. -- Polytropon From Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Nov 9, 2008, at 12:18 AM, Ian wrote: On Sun, 9 Nov 2008 13:10:46 Jeremy Chadwick wrote: On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote: Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? As far as I know, there is no such limit on the number of files/dirs inside of a directory. Thanks for that Jeremy. I didn't invent this structure, but I daresay I can either modify it or get the original writer to do that. I never really gave it a thought before now - it was the system I was given to work with and it's worked fine so far, except when I try to list the contents of the directory - that takes ages! All the folders are 7 digit numbers and we are up to approx. 0010500 entries (ie subdirs) so far. I guess it will just be a matter of experimenting to find the optimum number of sub-sub-directories per sub-directory :-/ On the issue of possible inode limitations, it may be of some reassurance (or alarm ;-)) to look at the output of df -i. This will tell you if you are close to any limit on inodes. Between that and your already well known counts of directories and rate of creation, you can gauge how much time you have to change your app. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 10:47:11AM +0100, Polytropon wrote: On Sun, 9 Nov 2008 10:35:21 +0100, Erik Trulsson [EMAIL PROTECTED] wrote: Note that this does not limit the number of files you can have in a single directory, since normal files do not contain hardlinks to the parent directory, but there are of course limits to the total number of files and directories you can have on a single filesystem based on how many inodes were created when the filesystem was first created. Maybe this sounds stupid, but... given that a file system can hold n entries. What happens when a program tries to create file number n + 1? I do ask this in order to explore if this could have been the reason for my massive data loss and UFS file system corruption. I haven't tested what actually happens, but what should happen is that the attempt to create file n+1 will simply fail with some appropriate error code (see open(2) or mkdir(2) for details.) It is certainly not supposed to cause any kind of files system corruption. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Jeremy Chadwick wrote: I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. With the implementation of UFS_DIRHASH the practical limit on the size of directories is now a great deal larger. In particular the slow down caused by linear search through the contents has been eliminated. See ffs(7). 10,000 files or sub-directories, whist not a particularly elegant setup, is actually not unworkable nowadays. As for the maximum number of subdirectories it is possible to create on UFS2 -- it is limited by the inode structure to a 16 bit quantity. % jot 10 1 | xargs mkdir -v [...] 32725 32726 32727 32728 32729 32730 3273mkdir: 32766: Too many links mkdir: 32767: Too many links mkdir: 32768: Too many links mkdir: 32769: Too many links mkdir: 32770: Too many links mkdir: 32771: Too many links [...] Which is 32768 - 2 for the '.' and '..' links. Trying to create too many subdirectories just results in mkdir failing: the filesystem itself is not damaged. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: UFS2 limits
Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? make sure your kernel is compiled with options UFS_DIRHASH or it will be slow the limit is 32765, just because link count is 2 bytes wide and each subdir adds two to base directory. you have to change to 2 level hierarchy. with files - i started creating empty files, it turned slow after about 32. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
With the implementation of UFS_DIRHASH the practical limit on the size of directories is now a great deal larger. In particular the slow down caused by linear search through the contents has been but - try making (by shell script for example) empty files. it creates it fast and rapidly slows at about 32 on my machine. i have UFS_DIRHASH ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Wojciech Puchar writes: the limit is 32765, just because link count is 2 bytes wide and each subdir adds two to base directory. you have to change to 2 level hierarchy. Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? Robert Huff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 11:02:07AM -0500, Robert Huff wrote: Wojciech Puchar writes: the limit is 32765, just because link count is 2 bytes wide and each subdir adds two to base directory. you have to change to 2 level hierarchy. Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? At this point, I think this topic should be relocated to the freebsd-fs mailing list. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Jeremy Chadwick([EMAIL PROTECTED])@2008.11.08 18:40:46 -0800: I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. On a related note there are filesystems that handle many files/directories very quickly. They use alternate tree data structures that behave quicker. ReiserFS is one of them. I believe XFS does quite well too. FreeBSD should have adapted XFS in addition to ZFS. ZFS is a resource monster. Shame, really. XFS is freely available in Linux for a number of years. Hammer, the new FS for FreeBSDs is available for DragonflyBSD. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 11:02:07AM -0500, Robert Huff wrote: Wojciech Puchar writes: the limit is 32765, just because link count is 2 bytes wide and each subdir adds two to base directory. you have to change to 2 level hierarchy. Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? It probably could be expanded to 32 bits if that was deemed useful. Doing that would of course require re-creating any existing filesystems since the on-disk format would change, which would be a PITA for users, but certainly possible. It is rare that anybody actually encounter this limit however. I would even say that if you have more than a couple of thousand entries in a single directory, then you are probably doing something wrong. Personally I cannot think of any situation where one would actually want (let alone need) as many as 3 or more subdirectories in a single directory. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 12:48:51PM -0500, Dan wrote: Erik Trulsson([EMAIL PROTECTED])@2008.11.09 17:53:14 +0100: Personally I cannot think of any situation where one would actually want (let alone need) as many as 3 or more subdirectories in a single directory. I've seen some Java apps that use the FS as the DB. Nothing wrong with that. I think an FS can be quite a good DB, if implemented well. This gives many data manipulation options with traditional FS tools, shell scripts, etc. Lets just say that there are reasons why the major database systems generally use their own methods to store and organize the data rather than rely on the file system for that. Besides, for most database applications I can think of, what you would need are lots of *files*, which do not have any special limitations other than the the total space and number of i-nodes on the filesystem. Even if you were using the FS as a DB I can't think of any good reason to need 3+ subdirectories in a single directory. Large Maildirs for postfix and qmail/Courier. Some people don't delete email at all. Again, that requires lots of files, not lots of subdirectories. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
If you really think HAMMER accomplishes the same goals as ZFS, you are sadly mistaken. it will be OK to achieve the goals it is advertised to achieve. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Erik Trulsson([EMAIL PROTECTED])@2008.11.09 17:53:14 +0100: Personally I cannot think of any situation where one would actually want (let alone need) as many as 3 or more subdirectories in a single directory. I've seen some Java apps that use the FS as the DB. Nothing wrong with that. I think an FS can be quite a good DB, if implemented well. This gives many data manipulation options with traditional FS tools, shell scripts, etc. Large Maildirs for postfix and qmail/Courier. Some people don't delete email at all. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 12:33:06PM -0600, Modulok wrote: Personally I cannot think of any situation where one would actually want (let alone need) as many as 3 or more subdirectories in a single directory. No one will ever need more than 640K of memory! Not quite the same thing. One major problem with having lots of entries in a directory is for humans using it (who have not become significantly faster or more capable over the recent decades.) Having lots of entries in a single directory is simply very unwieldy. There are is a reason why people invented hierarchichal files systems with directories and sub-directories, you know. For those situations where the directory is not intended to be looked at by a human, but only by programs, then there are more efficient ways of storing the data if you need that many entries. (A real database system, for example.) Besides, most (all?) of the situations where one might concievably want many entries in a single directory, what one would usually want is lots of files, not lots of sub-directories - once you start using sub-directories, you might as well use more than a single level of them. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 01:58:11PM -0500, Robert Huff wrote: Erik Trulsson writes: Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? It probably could be expanded to 32 bits if that was deemed useful. Doing that would of course require re-creating any existing filesystems since the on-disk format would change, which would be a PITA for users, but certainly possible. I seem to remember at least one case (3.x - 4.0 ) where a major version change had no upgrade path - to get the new stuff you had to reinstall. You are probably thinking of the 4.x - 5.x upgrade where you pretty much had to reinstall if you wanted to switch from UFS1 to UFS2. (But you could of course keep using UFS1 if you wanted.) But I agree there's no reason based on current evidence to do this. Thanks. Robert Huff -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
the limit is 32765, just because link count is 2 bytes wide and each subdir adds two to base directory. you have to change to 2 level hierarchy. Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? looking at /usr/include/ufs/ufs/dinode.h - i see int64_t di_spare[3]; and i have really no idea why time uses as much as 8+4 bytes like that: ufs_time_t di_mtime; /* 40: Last modified time. */ int32_t di_mtimensec; /* 64: Last modified time. */ i think it is not a problem to make link count 32-bit, and - why spare space are not just used for more direct/indirect blocks ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, 9 Nov 2008 22:46:18 Matthew Seaman wrote: Jeremy Chadwick wrote: I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. With the implementation of UFS_DIRHASH the practical limit on the size of directories is now a great deal larger. In particular the slow down caused by linear search through the contents has been eliminated. See ffs(7). 10,000 files or sub-directories, whist not a particularly elegant setup, is actually not unworkable nowadays. Well that's certainly been my experience so far. Still, I now know we will run into problems when we hit the 32,768 limit, so I'll start designing something better. Cheers, -- Ian gpg key: http://home.swiftdsl.com.au/~imoore/no-spam.asc signature.asc Description: This is a digitally signed message part.
Re: UFS2 limits
On Sun, Nov 09, 2008 at 11:09:17AM -0500, Dan wrote: Jeremy Chadwick([EMAIL PROTECTED])@2008.11.08 18:40:46 -0800: I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. On a related note there are filesystems that handle many files/directories very quickly. They use alternate tree data structures that behave quicker. ReiserFS is one of them. I believe XFS does quite well too. FreeBSD should have adapted XFS in addition to ZFS. ZFS is a resource monster. Shame, really. XFS is freely available in Linux for a number of years. Hammer, the new FS for FreeBSDs is available for DragonflyBSD. If you really think HAMMER accomplishes the same goals as ZFS, you are sadly mistaken. I have no idea about XFS. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Personally I cannot think of any situation where one would actually want (let alone need) as many as 3 or more subdirectories in a single directory. No one will ever need more than 640K of memory! Be careful. -Modulok- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
number of years. Hammer, the new FS for FreeBSDs is available for DragonflyBSD. i would like to see final (now still beta) version of hammer in action. it's ADVERTISED features are great. but ZFS features was (and are) ADVERTISED great too while we see the result. Hammer would be great if it will be as advertised :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Erik Trulsson([EMAIL PROTECTED])@2008.11.09 20:54:14 +0100: Besides, for most database applications I can think of, what you would need are lots of *files*, which do not have any special limitations other than the the total space and number of i-nodes on the filesystem. Even if you were using the FS as a DB I can't think of any good reason to need 3+ subdirectories in a single directory. Large Maildirs for postfix and qmail/Courier. Some people don't delete email at all. Again, that requires lots of files, not lots of subdirectories. I agree, I misread the OP. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 9, 2008 at 2:55 PM, Erik Trulsson [EMAIL PROTECTED]wrote: On Sun, Nov 09, 2008 at 01:58:11PM -0500, Robert Huff wrote: Erik Trulsson writes: Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? It probably could be expanded to 32 bits if that was deemed useful. Doing that would of course require re-creating any existing filesystems since the on-disk format would change, which would be a PITA for users, but certainly possible. I seem to remember at least one case (3.x - 4.0 ) where a major version change had no upgrade path - to get the new stuff you had to reinstall. You are probably thinking of the 4.x - 5.x upgrade where you pretty much had to reinstall if you wanted to switch from UFS1 to UFS2. (But you could of course keep using UFS1 if you wanted.) But I agree there's no reason based on current evidence to do this. Thanks. Robert Huff -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Kind of hard to get XFS in freeBSD with it being a dead filesystem that is no longer being developed, probably to port it it would need a lot of code changes. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
Erik Trulsson writes: Question (for anyone who has an informed opinion): If there any technical reason that couldn't be expanded to 32 bits? Or is it possible but not done for historical or policy reasons, and if so what are they? It probably could be expanded to 32 bits if that was deemed useful. Doing that would of course require re-creating any existing filesystems since the on-disk format would change, which would be a PITA for users, but certainly possible. I seem to remember at least one case (3.x - 4.0 ) where a major version change had no upgrade path - to get the new stuff you had to reinstall. But I agree there's no reason based on current evidence to do this. Thanks. Robert Huff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
UFS2 limits
Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? What about ZFS? At some point I'll have to re-arrange things so that I have a deeper directory structure, just wondering when I'll hit the limit so I can plan in advance :-) Cheers, Ian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 limits
On Sun, Nov 09, 2008 at 01:40:51AM +, [EMAIL PROTECTED] wrote: Hi, I have a FreeBSD server that has about 10,500 subdirectories within a single directory. This number will keep rising and I assume UFS2 has a limit to the number of sub-directories in a single directory - can anyone tell me what it is? As far as I know, there is no such limit on the number of files/dirs inside of a directory. I don't want to change the topic of discussion, but I *highly* recommend you ***stop*** whatever it is you're doing that is creating such a directory structure. Software which has to iterate through that directory using opendir() and readdir() will get slower and slower as time goes on. If this is something you've written or have control over or can work with engineers in regards to, I recommend you change your directory naming scheme to have separate subdirectories with the first 2 or 3 letters of the directory you wish to create. E.g.: /some/place/00/00ilikezeros/* /some/place/01/01binaryheaven/* /some/place/aa/aardvarks/* /some/place/ab/abuse/* /some/place/ac/actuary/* ... /some/place/xy/xylophones/* You get the point. Traversing this structure is much more efficient, and requires very little code change on your part. Those who run nameservers that host many zones, for example, use this structure to ensure the daemon doesn't take 32498231 years to start up. What about ZFS? At some point I'll have to re-arrange things so that I have a deeper directory structure, just wondering when I'll hit the limit so I can plan in advance :-) What baffles me is why you're looking at this problem from a how can the filesystem solve this engineering mistake I made for me standpoint, rather than how can I solve this engineering mistake I made so that it doesn't impact the filesystem. Very strange. Sometimes looking at things in a different light makes all the difference. Hope this helps. P.S. -- I hope this mail makes it to you, because your From line is [EMAIL PROTECTED] (I'll be surprised if your account name really is that!). -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
