Re: Max files in unix folder from PIL process
Hi. I am creating a python application that uses PIL to generate thumbnails and sized images. It is beginning to look the volume of images will be large. This has got me to thinking. Is there a number that Unix can handle in a single directory. I am using FreeBSD4.x at the moment. I am thinking the number could be as high 500,000 images in a single directory but more likely in the range of 6,000 to 30,000 for most. I did not want to store these in Postgres. I will most likely to break these into directories by size ie. thumbnail, small, medium, large, etc. . That will at least take it down by a factor of the number of sizes used but still the possibility of a very large number (maximum to perhaps 100,000 or more) There is really no other way that I can think of to categorize these at the moment. Should this pose a problem on the filesystem? How will it affect the use of Unix tools? Will there be access problems that affect speed? This is unchartered territory for me so hope someone who has been there, done that can provide some of what they learned from experience. Many thanks. Regards, David ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Max files in unix folder from PIL process
On Mon, 28 Mar 2005 21:25:28 -0400, David Pratt [EMAIL PROTECTED] wrote: Hi. I am creating a python application that uses PIL to generate thumbnails and sized images. It is beginning to look the volume of images will be large. This has got me to thinking. Is there a number that Unix can handle in a single directory. I am using FreeBSD4.x at the moment. I am thinking the number could be as high 500,000 images in a single directory but more likely in the range of 6,000 to 30,000 for most. I did not want to store these in Postgres. I will most likely to break these into directories by size ie. thumbnail, small, medium, large, etc. . That will at least take it down by a factor of the number of sizes used but still the possibility of a very large number (maximum to perhaps 100,000 or more) There is really no other way that I can think of to categorize these at the moment. Should this pose a problem on the filesystem? How will it affect the use of Unix tools? Will there be access problems that affect speed? This is unchartered territory for me so hope someone who has been there, done that can provide some of what they learned from experience. Many thanks. Regards, David ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] I've had apps that had over 800k subdirectories and files. As long as you know the filename, performance will not be an issue. However, if you don't know the full path, then you may want to either redesign the app or consider storing everything in the DB. The apps that I've seen had the full path stored in the database, and the image on a file server. I've also had apps that stored everything in the database, and that turned out to be a nightmare. Eventually, we moved everything from the DB to the filesystem. Corey ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Max files in unix folder from PIL process
Hi Corey. Thank you for your reply. 800K is pretty significant. Yes, the reason I want to use the filesystem is to avoid the speed problems that come from storing images in the database. I can see there really being no limit when it comes to spreading the numbers thinner but most concerned about so many in a single directory. References to the images will come from the database. Do you don't think I should have any problem with 150 - 175 K in a single directory - no subdirectories - (just 175K jpg image files of a particular size)? I guess you definitely don't want to do an ls. Would that crap out the server? Regards, David On Monday, March 28, 2005, at 09:44 PM, Corey Brune wrote: I've had apps that had over 800k subdirectories and files. As long as you know the filename, performance will not be an issue. However, if you don't know the full path, then you may want to either redesign the app or consider storing everything in the DB. The apps that I've seen had the full path stored in the database, and the image on a file server. I've also had apps that stored everything in the database, and that turned out to be a nightmare. Eventually, we moved everything from the DB to the filesystem. Corey ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Max files in unix folder from PIL process
On Tuesday 29 March 2005 03:25, David Pratt wrote: Hi. I am creating a python application that uses PIL to generate thumbnails and sized images. It is beginning to look the volume of images will be large. This has got me to thinking. Is there a number that Unix can handle in a single directory. I am using FreeBSD4.x at the moment. I am thinking the number could be as high 500,000 images in a single directory but more likely in the range of 6,000 to 30,000 for most. I did not want to store these in Postgres. I will most likely to break these into directories by size ie. thumbnail, small, medium, large, etc. . That will at least take it down by a factor of the number of sizes used but still the possibility of a very large number (maximum to perhaps 100,000 or more) There is really no other way that I can think of to categorize these at the moment. Should this pose a problem on the filesystem? How will it affect the use of Unix tools? Will there be access problems that affect speed? This is unchartered territory for me so hope someone who has been there, done that can provide some of what they learned from experience. Many thanks. I'd say hardcode a split-up at about ~5000 images per dir and renice if needed. Test with python whether reading one huge dir or reading and concatting several smaller (say 5000 files) dirs works better. Apparantly you got the pr0n, err, data, .. so you're in a position to test just what you're wondering about. If you want speed, stay away from databases/sql I think. I only have experience with my own pr0n, err, data, and its far less than what you want to deal with (pil works great though). My EUR 0.02 Dan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
