Thanks a lot Dave. This gives me some great ideas. I appreciated the input.
-Brian On Feb 6, 8:05 pm, Dave <[email protected]> wrote: > Hey Brian, > > My last project was for a SASS company that dealt with graphics files > (from 100kb to 150mb each!). Storage when I left was around 4tb of > files, 2tb of on-line and 2tb of near-line. > > On a very basic level, we stored pointers in the DB. There was an > abstraction around this called "the asset system". > > It had the concept of "assets" (files), which were stored in "asset > stores" (folders). > > Assets stores had both a UNC path to the folder, and public URI if > applicable. Assets contained pointers to the file name (generation of > these file names took a few attempts to get right due to concurrency > issues, ie, 20 files uploaded at the same time) > > Stores were configured with a maximum size, allocation priority, etc. > They were organised into logical management units called "asset > farms", (per client, per file type, etc). > > Adding more storage was simply a case of adding a new Asset Store & > allocating it to an Asset Farm, this Asset Store could reside on any > server, SAN, NAS, etc because to the software, it was really just a > UNC path to a folder). > > We didn't worry about ZIP/compressing uploaded files, as storage it > cheap these days. > > We implemented arching to near-line, purging after N years, etc. > These sorts of issues become a lot easer to deal with when you each > file had a corresponding database record with metadata. > > The biggest tip I can give is to come up with a nice API for dealing > with your "storage system abstraction", the rest becomes really easy. > > I wouldn't store images in the DB as you are really just wasting DB > resources (which are expensive) for a problem that can be solved at > the app level. > > Dave > > On Feb 7, 7:56 am, "Brian H." <[email protected]> wrote: > > > Sorry that this is an off-topic thread, but I know of no other CF > > community resource for general questions. > > > I have written a large application for law firms to manage their files > > and communications. They can upload files (pdfs, docs, etc) and > > generate then using the system. I have also developed a fully > > integrated communication component supporting online email and fax, so > > every email attachment that they send or receive gets stored as well. > > > So basically I have a system with tons of files (about 2GB for this > > one law firm). Currently I store these files on disk, and keep a > > reference to them in the database. This has been fine for this one > > client, but they now wish to take this application to market and build > > it out under a SaaS model, so some transitioning is going to happen. > > I am wondering how you guys (and gals) manage files in your > > applications? Our target is to build this system out to support 200 > > law firms, at which point we would rebuild the system with SaaS in > > mind from the start. 400GB of file storage does not seem practical in > > this case. > > > So how do you folks handle things? > > > Do you typically ZIP uploaded files? > > > Do you use OS directory compression on file storage folders? That > > would be an easy fix requiring no code adjustment. > > > Do you implement archiving in your application? If so, how do you go > > about this? > > > Do you store files in the DB? Is this really practical when storage > > might be 100GB or more? I would think that DB backups would be hell. > > > Any input would be appreciated. > > > -Brian -- You received this message because you are subscribed to Mach-II for CFML list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/mach-ii-for-coldfusion?hl=en SVN: http://greatbiztoolsllc.svn.cvsdude.com/mach-ii/ Wiki / Documentation / Tickets: http://greatbiztoolsllc.trac.cvsdude.com/mach-ii/
