On Tuesday 29 January 2002 12:40, Sinisa Milivojevic wrote: > Tod Harter writes: > > Hi Guys > > > > I've been using MySQL for a few years now, but always on medium sized > > projects. I think the biggest thing I ever did has maybe 200k records in > > it. > > > > Now all of a sudden I have a client with a requirement for a database > > that we estimate will be in the range of 1 to 4 TERABYTES of data... Now > > this data is largely static (though there will be a few updates and some > > insertions) non-transactional, and needs to be searched on several > > indexes. Furthermore its image data, plus metadata (which the searching > > is done on). > > > > Anyone have experience with this kind of application? Is MySQL going to > > handle it? What are the upwards limits on scalability? Ideally from a > > system design perspective I'd like to have ALL the data in the database > > itself and in a single set of tables. Unfortunately the images themselves > > are up to 10 megs each. Is it even POSSI BLE to use blobs in that > > context? I know someone is going to suggest putting the image data > > outside the database, but for various reasons I consider that a last > > resort, not the least of which is security requirements are so high that > > running other protocols to access data thats outside tables is > > problematic. > > > > I'd love to hear about other's experience in this area. Thanks ahead of > > time > > > > :o). > > Hi! > > A number of our registered customers has database of that size. > > But in order to optimise things, I do not think it is wise to store > such huge images in tables.
What would be the performance considerations? Considering it theoretically if I (for instance) set things up so that I had an ftp server for the images and stored filenames in a column then as far as raw data transfer efficiency its tcp either way. I guess the question becomes one of how does the database deal with blobs internally? Given that the data is images its essentially opaque data to the RDBMS (ie there would never be any reason to search the blob columns themselves, just recover them based on searches of other indexes). I guess the other related question with indexes is how big a deal is it to have such huge indexes? In a practical sense its one data set, so I really would like to have indexes covering the whole thing. The number of records runs up into the low millions. Queries will need to be done on maybe up to 4 or 5 columns in that data set. Can I expect a query like that to complete in a time frame of under 1 minute? What kind of hardware would be recommended to get that sort of performance? We will be building out the entire system, so essentially its up to us to define that. I'd considered something in the order of a top of the line dual processor Pentium 4 class system with a couple of SAN boxes and fibre channel. I expect its no challenge to throw a few gigs of ram at it, but naturally there is NO way to put enough ram in a box to even come close to holding complete indexes... --------------------------------------------------------------------- Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
