On Wed, 2004-11-24 at 07:37, Christopher Schmidt wrote: > On Wed, Nov 24, 2004 at 07:28:05AM -0500, Fred wrote: > > If there resources are so limited that they have to worry about 20K or > > 30K downloads, they really should seriously consider upgrading their > > computers!!!! > > LiveJournal.com hosts four racks of hardware, including 7 different > database clusters and something along the lines of 60 web nodes, yet for > exporting data via a web interface, you are limited to only one month at > a time - which is approximately this size. > > Load can be considerable from heavy database queries, which I assume > exporting data in this manner is. Anyone who's worked with databases for > a while can tell you that it's not always best to allow "SELECT * FROM > foo WHERE" with no limit: there can be a lot of data in there.
Of course. For one particular client who wanted real time stats updates for his Affiliate module, I had to tell him that's unrealistic. So, stats are updated nightly instead. But that's all a matter of database design. I cache certain data in tables that would otherwise overwhelm a SELECT * query. Proper use of indices also becomes critical. Affiliates do demand a lot of data viewed in different ways, and it can be a challenge at times. > There's also the fact that you may be dealing with some people that have > much higher numbers of transactions: dozens, or more, per day. To select > 1000 records and all their associated data is not a simple task. There is that for sure, considering this is a business account. > Limiting by actual transactions is going to confuse customers, but they > can limit by a time frame without too much problems on either the "too > much load" or the "too confusing to users" sides. > > Not neccesarily the case, but "buying more computers" isn't always the > solution. Of course not. But smart DB design. caching often-accessed data, and distributed loading techniques, etc. can make all the difference. Just that I expect large banks such as Bank of America, etc. to be able to afford the expertise and resources to do this. I've worked in Credit Card processing for a while in the past, and the sheer data requirements are *enormous*. What it terabytes per minute, or terabytes per second? I forget. The resources involved with just a single credit card purchase is mind-boggling, considering it has to go through the merchant's account through a central routing system like First Data to your individual account at your financial institution, and back again, all within seconds. And yet it works so darn well we don't even notice it -- it is technology that has folded almost seamlessly into the background. Same can be said for stock trading, even more so. Thousands of dollars are flipped between trader's accounts within milliseconds. I've gotten into long nasty debates over stock trading -- many people are unwilling to see this as the zero-sum game for what it is. When you purchase 1000 shares of CSCO at $20 a share, $20,000 goes from your account to the accounts of the traders/investors your bought the shares from. When you then sell CSCO later at $15 a share in panic, you have to find -- or the system does -- $15,000 from the accounts of other traders willing to deal with you. Money then goes from their accounts to yours. And the hardnosed fact is the time scales of how long you hold onto CSCO does not change the zero-sum nature. I've seen stocks of other companies plummet from hundreds of dollars a share to just mere pennies. Or so-called "blue-chip" stocks crumble into dust. Trading is not for the faint of heart, and it would be sheer folly to base one's retirement plans off the stock market, though many do. Few have the time or the skills to watch their money that closely, and good luck if you think a financial adviser will do it for you. Well, I'm getting extremely divergent from the topic here, so I'd better quit now while I'm ahead... :-/ -- Fred -- [EMAIL PROTECTED] -- place "[hey]" in your subject. The mass of humans on planet Earth -- regard them as the ebbing seas in the winds of change. They ebb, they flow, they know not where to go. _______________________________________________ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
