Re: managing user uploads best practices
Totally agree with everything brien says below. I also run a web site with a large number of media objects. I've been involved in filing system and media projects for many years both in research and production capacity. The slight increase in complexity of the solution to maintain the media objects outside the database is completely offset by the huge improvement in performance of the database. Also it is particularly worth stressing the backup advantages. If you have 10,000 users (not a large number these days) each with 5 images and/or one podcast you are going to have very, very large monolithic database backups which are either going to be in a binary form and thus very opaque or simply huge as sql dumps unless you separate the binary data. Another advantage is that it allows you to have a very scalable media server. The media server does not have to participate in the session. You can have a number of independent media servers who simply download files identified by an ID. The media server(s) don't have to be on the same box, platform or even in the same building. The issues of clustering simply go away (a media download is a single operation and thus does not benefit from clustering.) If you are running a large number of concurrent downloads, you only need a trivial load balancing front end server to direct requests at the media servers. If you require authentication to restrict access to the media then you may have a bit more work to do but it is not too difficult. So if you have a site with a lot of media, you can tune the application component to be small and responsive and the media delivery components to be simple and fast and optimizied for continuous download. brien colwell wrote: On the topic of DB versus filesystem for media, I prefer storing media in a filesystem and meta data in a DB. The advantages of storing large binary files outside of the DB are * Reduce contention in the DB -- it's doing so much * You have more control with a filesystem where your data goes -- e.g. you can have a disk for one type of data, and another disk for a different type * Filesystems are optimized to stream out data ... which I think makes sense That first point is really key. If you're serving data to a UI from the same database you're storing large files into, the whole experience is going to suffer. The best article I've read on this topic is http://mysqldatabaseadministration.blogspot.com/2008/01/i-will-not-blob.html The main scale issue I'm careful of is how many files I put per directory. For large amounts of files, I sometimes link directories into trees. I might be a little paranoid about this, but it seems to scale well. In terms of backup and security, I think a FS is easier to manage -- e.g. chmod and rsync on a cron job -- though you'll have to write more code and scripts. You also have to be careful of things like shell injection if you're running shell commands from your webserver. Anyway, hope that gives some help. I was also confused with this when I started writing DB backed apps. On Feb 9, 2008 4:58 PM, <[EMAIL PROTECTED]> wrote: Thanks for the suggestions! I like using the database for storage for it's easy maintenance and security, but marshalling a lot of binary data (like a large image library) adds a bit of overhead to the application. I'll look into pursuing a storage directory external to the webapp. -- Original message -- From: "Johnny Kewl" <[EMAIL PROTECTED]> --- HARBOR: http://coolharbor.100free.com/index.htm The most powerful application server on earth. The only real POJO Application Server. Making the Java dream come true. --- - Original Message - From: <[EMAIL PROTECTED]> To: Sent: Friday, February 08, 2008 11:13 PM Subject: managing user uploads best practices Yes... outside. Its been a long long time now, vaguely remember struggling with Apache uploader then eventually getting all to work... Anyway... what I did is store the files in an Apache httpD folder, so I could spy on the uploads, and they available for viewing again. And whats cool is because Apache is also the load balancer in my case... can have lots of TC's doing their thing. I was making a kind of wiki thing for an estate agency... thats how I did it way back when... What's the current wisdom on managing user uploaded files to a web app that's deployed via a WAR? In other words, when the WAR is updated, the directory containing uploaded files would be wiped out. Do people save uploaded files outside of the web app root directory? Security issues with this? Do people not use auto-expanding WAR files and manage the deployment by hand? Do you not include the directory f
Re: managing user uploads best practices
On the topic of DB versus filesystem for media, I prefer storing media in a filesystem and meta data in a DB. The advantages of storing large binary files outside of the DB are * Reduce contention in the DB -- it's doing so much * You have more control with a filesystem where your data goes -- e.g. you can have a disk for one type of data, and another disk for a different type * Filesystems are optimized to stream out data ... which I think makes sense That first point is really key. If you're serving data to a UI from the same database you're storing large files into, the whole experience is going to suffer. The best article I've read on this topic is http://mysqldatabaseadministration.blogspot.com/2008/01/i-will-not-blob.html The main scale issue I'm careful of is how many files I put per directory. For large amounts of files, I sometimes link directories into trees. I might be a little paranoid about this, but it seems to scale well. In terms of backup and security, I think a FS is easier to manage -- e.g. chmod and rsync on a cron job -- though you'll have to write more code and scripts. You also have to be careful of things like shell injection if you're running shell commands from your webserver. Anyway, hope that gives some help. I was also confused with this when I started writing DB backed apps. On Feb 9, 2008 4:58 PM, <[EMAIL PROTECTED]> wrote: > Thanks for the suggestions! > > I like using the database for storage for it's easy maintenance and security, > but marshalling a lot of binary data (like a large image library) adds a bit > of overhead to the application. > > I'll look into pursuing a storage directory external to the webapp. > > > > -- Original message -- > From: "Johnny Kewl" <[EMAIL PROTECTED]> > > > > --- > > HARBOR: http://coolharbor.100free.com/index.htm > > The most powerful application server on earth. > > The only real POJO Application Server. > > Making the Java dream come true. > > --- > > ----- Original Message - > > From: <[EMAIL PROTECTED]> > > To: > > Sent: Friday, February 08, 2008 11:13 PM > > Subject: managing user uploads best practices > > > > Yes... outside. > > Its been a long long time now, vaguely remember struggling with Apache > > uploader then eventually getting all to work... > > Anyway... what I did is store the files in an Apache httpD folder, so I > > could spy on the uploads, and they available for viewing again. > > And whats cool is because Apache is also the load balancer in my case... can > > have lots of TC's doing their thing. > > I was making a kind of wiki thing for an estate agency... thats how I did > > it way back when... > > > > > > > What's the current wisdom on managing user uploaded files to a web app > > > that's > > > deployed via a WAR? > > > > > > In other words, when the WAR is updated, the directory containing uploaded > > > files > > > would be wiped out. > > > > > > Do people save uploaded files outside of the web app root directory? > > > Security > > > issues with this? > > > Do people not use auto-expanding WAR files and manage the deployment by > > > hand? > > > Do you not include the directory for uploaded files in the WAR (but create > > > it at > > > runtime) and then trust that the expanded WAR won't overwrite it on > > > deployment? > > > > > > Any pointers greatly appreciated! > > > > > > - > > > To start a new topic, e-mail: users@tomcat.apache.org > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > - > > To start a new topic, e-mail: users@tomcat.apache.org > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > - > To start a new topic, e-mail: users@tomcat.apache.org > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: managing user uploads best practices
Thanks for the suggestions! I like using the database for storage for it's easy maintenance and security, but marshalling a lot of binary data (like a large image library) adds a bit of overhead to the application. I'll look into pursuing a storage directory external to the webapp. -- Original message -- From: "Johnny Kewl" <[EMAIL PROTECTED]> > > --- > HARBOR: http://coolharbor.100free.com/index.htm > The most powerful application server on earth. > The only real POJO Application Server. > Making the Java dream come true. > --- > - Original Message - > From: <[EMAIL PROTECTED]> > To: > Sent: Friday, February 08, 2008 11:13 PM > Subject: managing user uploads best practices > > Yes... outside. > Its been a long long time now, vaguely remember struggling with Apache > uploader then eventually getting all to work... > Anyway... what I did is store the files in an Apache httpD folder, so I > could spy on the uploads, and they available for viewing again. > And whats cool is because Apache is also the load balancer in my case... can > have lots of TC's doing their thing. > I was making a kind of wiki thing for an estate agency... thats how I did > it way back when... > > > > What's the current wisdom on managing user uploaded files to a web app > > that's > > deployed via a WAR? > > > > In other words, when the WAR is updated, the directory containing uploaded > > files > > would be wiped out. > > > > Do people save uploaded files outside of the web app root directory? > > Security > > issues with this? > > Do people not use auto-expanding WAR files and manage the deployment by > > hand? > > Do you not include the directory for uploaded files in the WAR (but create > > it at > > runtime) and then trust that the expanded WAR won't overwrite it on > > deployment? > > > > Any pointers greatly appreciated! > > > > - > > To start a new topic, e-mail: users@tomcat.apache.org > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > - > To start a new topic, e-mail: users@tomcat.apache.org > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: managing user uploads best practices
--- HARBOR: http://coolharbor.100free.com/index.htm The most powerful application server on earth. The only real POJO Application Server. Making the Java dream come true. --- - Original Message - From: <[EMAIL PROTECTED]> To: Sent: Friday, February 08, 2008 11:13 PM Subject: managing user uploads best practices Yes... outside. Its been a long long time now, vaguely remember struggling with Apache uploader then eventually getting all to work... Anyway... what I did is store the files in an Apache httpD folder, so I could spy on the uploads, and they available for viewing again. And whats cool is because Apache is also the load balancer in my case... can have lots of TC's doing their thing. I was making a kind of wiki thing for an estate agency... thats how I did it way back when... What's the current wisdom on managing user uploaded files to a web app that's deployed via a WAR? In other words, when the WAR is updated, the directory containing uploaded files would be wiped out. Do people save uploaded files outside of the web app root directory? Security issues with this? Do people not use auto-expanding WAR files and manage the deployment by hand? Do you not include the directory for uploaded files in the WAR (but create it at runtime) and then trust that the expanded WAR won't overwrite it on deployment? Any pointers greatly appreciated! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: managing user uploads best practices
I certainly don't speak for everyone, but in the past I've either provided an external folder for upload storage or a database. As far as security goes, I don't see these being much if any different than storing the files inside the webapp. In some ways you may get better access control as the servlet serving them can be customized to provide very fine grained access. --David [EMAIL PROTECTED] wrote: What's the current wisdom on managing user uploaded files to a web app that's deployed via a WAR? In other words, when the WAR is updated, the directory containing uploaded files would be wiped out. Do people save uploaded files outside of the web app root directory? Security issues with this? Do people not use auto-expanding WAR files and manage the deployment by hand? Do you not include the directory for uploaded files in the WAR (but create it at runtime) and then trust that the expanded WAR won't overwrite it on deployment? Any pointers greatly appreciated! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
managing user uploads best practices
What's the current wisdom on managing user uploaded files to a web app that's deployed via a WAR? In other words, when the WAR is updated, the directory containing uploaded files would be wiped out. Do people save uploaded files outside of the web app root directory? Security issues with this? Do people not use auto-expanding WAR files and manage the deployment by hand? Do you not include the directory for uploaded files in the WAR (but create it at runtime) and then trust that the expanded WAR won't overwrite it on deployment? Any pointers greatly appreciated! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
