RE: OT -- 90-day limits in the financial world for downloading your data.
On Mon, 2004-11-22 at 09:12, Tilly, Lawrence wrote: It could also be data access time. Not sure what software they're using, but while you're doing your search you're probably tying up web threads, worker threads in a JVM (assuming java-based application server), database connections and cycles on the database machine. By limiting that to 90-day chunks they probably have a pretty good idea of the resources you'll have tied up. If they let people pull unlimited requests, it could have some pretty negative impact on their service. That would not serve that customer well, nor would it look good for all the other people trying to use it and getting poor performance. If there resources are so limited that they have to worry about 20K or 30K downloads, they really should seriously consider upgrading their computers -- Fred -- [EMAIL PROTECTED] -- place [hey] in your subject. The mass of humans on planet Earth -- regard them as the ebbing seas in the winds of change. They ebb, they flow, they know not where to go. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
On Wed, Nov 24, 2004 at 07:28:05AM -0500, Fred wrote: If there resources are so limited that they have to worry about 20K or 30K downloads, they really should seriously consider upgrading their computers LiveJournal.com hosts four racks of hardware, including 7 different database clusters and something along the lines of 60 web nodes, yet for exporting data via a web interface, you are limited to only one month at a time - which is approximately this size. Load can be considerable from heavy database queries, which I assume exporting data in this manner is. Anyone who's worked with databases for a while can tell you that it's not always best to allow SELECT * FROM foo WHERE with no limit: there can be a lot of data in there. There's also the fact that you may be dealing with some people that have much higher numbers of transactions: dozens, or more, per day. To select 1000 records and all their associated data is not a simple task. Limiting by actual transactions is going to confuse customers, but they can limit by a time frame without too much problems on either the too much load or the too confusing to users sides. Not neccesarily the case, but buying more computers isn't always the solution. -- Christopher SChmidt [EMAIL PROTECTED] pgppY0Vh80rz6.pgp Description: PGP signature
Re: OT -- 90-day limits in the financial world for downloading your data.
On Wed, 2004-11-24 at 07:37, Christopher Schmidt wrote: On Wed, Nov 24, 2004 at 07:28:05AM -0500, Fred wrote: If there resources are so limited that they have to worry about 20K or 30K downloads, they really should seriously consider upgrading their computers LiveJournal.com hosts four racks of hardware, including 7 different database clusters and something along the lines of 60 web nodes, yet for exporting data via a web interface, you are limited to only one month at a time - which is approximately this size. Load can be considerable from heavy database queries, which I assume exporting data in this manner is. Anyone who's worked with databases for a while can tell you that it's not always best to allow SELECT * FROM foo WHERE with no limit: there can be a lot of data in there. Of course. For one particular client who wanted real time stats updates for his Affiliate module, I had to tell him that's unrealistic. So, stats are updated nightly instead. But that's all a matter of database design. I cache certain data in tables that would otherwise overwhelm a SELECT * query. Proper use of indices also becomes critical. Affiliates do demand a lot of data viewed in different ways, and it can be a challenge at times. There's also the fact that you may be dealing with some people that have much higher numbers of transactions: dozens, or more, per day. To select 1000 records and all their associated data is not a simple task. There is that for sure, considering this is a business account. Limiting by actual transactions is going to confuse customers, but they can limit by a time frame without too much problems on either the too much load or the too confusing to users sides. Not neccesarily the case, but buying more computers isn't always the solution. Of course not. But smart DB design. caching often-accessed data, and distributed loading techniques, etc. can make all the difference. Just that I expect large banks such as Bank of America, etc. to be able to afford the expertise and resources to do this. I've worked in Credit Card processing for a while in the past, and the sheer data requirements are *enormous*. What it terabytes per minute, or terabytes per second? I forget. The resources involved with just a single credit card purchase is mind-boggling, considering it has to go through the merchant's account through a central routing system like First Data to your individual account at your financial institution, and back again, all within seconds. And yet it works so darn well we don't even notice it -- it is technology that has folded almost seamlessly into the background. Same can be said for stock trading, even more so. Thousands of dollars are flipped between trader's accounts within milliseconds. I've gotten into long nasty debates over stock trading -- many people are unwilling to see this as the zero-sum game for what it is. When you purchase 1000 shares of CSCO at $20 a share, $20,000 goes from your account to the accounts of the traders/investors your bought the shares from. When you then sell CSCO later at $15 a share in panic, you have to find -- or the system does -- $15,000 from the accounts of other traders willing to deal with you. Money then goes from their accounts to yours. And the hardnosed fact is the time scales of how long you hold onto CSCO does not change the zero-sum nature. I've seen stocks of other companies plummet from hundreds of dollars a share to just mere pennies. Or so-called blue-chip stocks crumble into dust. Trading is not for the faint of heart, and it would be sheer folly to base one's retirement plans off the stock market, though many do. Few have the time or the skills to watch their money that closely, and good luck if you think a financial adviser will do it for you. Well, I'm getting extremely divergent from the topic here, so I'd better quit now while I'm ahead... :-/ -- Fred -- [EMAIL PROTECTED] -- place [hey] in your subject. The mass of humans on planet Earth -- regard them as the ebbing seas in the winds of change. They ebb, they flow, they know not where to go. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
On Nov 24, 2004, at 07:28, Fred wrote: If there resources are so limited that they have to worry about 20K or 30K downloads, they really should seriously consider upgrading their computers Ah, mergeritis. See, your local bank with 10,000 customers might be OK with keeping a meg of data online for you and sifting through it for what you want to see. That's ten gigs of reliable online storage - not too bad. Then your bank gets bought by another bank which gets bought by another bank until there are three banks in the country, each with a hundred million customers. Then the bank has a hundred terabyte database to keep online and search through. Others have mentioned the follies of doing this with current technology - it's expensive and very hard. So they don't. Then they can offer better rates than the local banks because they've just axed the IT/customer service budget. So more people dump the local banks because all they see is rates. The government refers to this as productivity. -Bill Bill McGonigle, Owner Work: 603.448.4440 BFC Computing, LLC Home: 603.448.1668 [EMAIL PROTECTED] Cell: 603.252.2606 http://www.bfccomputing.com/Text: [EMAIL PROTECTED] AIM: wpmcgonigleSkype: bill_mcgonigle ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
Systems are a major part of the equation on 'Wall St.'. The systems are immense, and viewed as a tool to control costs, and increase profits. There are many influencers in the equation: employees, regulators, customers, partners, competitors etc. If there was a positive cost/benefit analysis to support it, you would get what you want. My previous comments may seem like those of a conspiracy theorist, but actually, my views come from roughly 10 years of experience training on Wall St. and working at the biggest financial institutions in Boston. Put more simply, money is the name of the game, and it's the only game in town. Customer convenience and service is provided as a function of some cost/benefit analysis. -- FREePHILE We are 'Open' for Business Free and Open Source Software http://www.freephile.com (978) 270-2425 If you are smart enough to know that you're not smart enough to be an Engineer, then you're in Business. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
On Wed, 2004-11-24 at 10:57, Bill McGonigle wrote: ... See, your local bank with 10,000 customers might be OK with keeping a meg of data online for you and sifting through it for what you want to see. That's ten gigs of reliable online storage - not too bad. Then your bank gets bought by another bank which gets bought by another bank until there are three banks in the country, each with a hundred million customers. Yep. This is exactly what happened. My accounts were initially at *BankBoston* -- remember them? Then they got bought out or merged with Fleet, and now Fleet has been consumed by Bank of America. I find this all very annoying, but what can I do? I suppose I can close down all my accounts and sign up with a local bank -- only to see them bought out by Bank of America or another Really Hugh Institution a year later. Sigh. Maybe I should go offshore and set up a bank for the rest of us. Oh, but the IRS would just love that! :-) -- Fred -- [EMAIL PROTECTED] -- place [hey] in your subject. The mass of humans on planet Earth -- regard them as the ebbing seas in the winds of change. They ebb, they flow, they know not where to go. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
RE: OT -- 90-day limits in the financial world for downloading your data.
It could also be data access time. Not sure what software they're using, but while you're doing your search you're probably tying up web threads, worker threads in a JVM (assuming java-based application server), database connections and cycles on the database machine. By limiting that to 90-day chunks they probably have a pretty good idea of the resources you'll have tied up. If they let people pull unlimited requests, it could have some pretty negative impact on their service. That would not serve that customer well, nor would it look good for all the other people trying to use it and getting poor performance. -L -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Greg Rundlett Sent: Sunday, November 21, 2004 10:08 PM To: Fred Cc: Bill Mullen; GNHLUG Subject: Re: OT -- 90-day limits in the financial world for downloading your data. Fred wrote: On Sat, 2004-11-20 at 12:17, Bill Mullen wrote: I suspect that the real issue here is merely one of storage space; by setting a fixed period for which they will make data available (last 90 days, last 3 statement periods, whatever), they can move enough transactions out of the database to keep up with the new transactions being added, all while keeping their online storage capacity fairly static and predictable. It's not that even -- online, a full 18 months of data is available; I am only allowed to access it 90 days at a time. The 90-day window can be *anywhere* in the total dataset; I just can't pull more than a 90-day chunk of it at any given time. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
I'm guessing this is the real reason... Original message Date: Sat, 20 Nov 2004 12:17:09 -0500 From: Bill Mullen [EMAIL PROTECTED] Years ago, before online banking existed, [...] We had the same policy then for the account history data that we made available to the tellers over the online system - current statement period plus the last two, that's it; everything beyond that was archived. Disk packs aren't cheap. :) -- Might even be rooted previous to automation, think about how many moon of account history data would be keep conveniently accessible using papyrus based technology... That's the thing about those conservative bankers, once they figure out how to do something they are loathe to change. -Bruce McCulley ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
Fred wrote: On Sat, 2004-11-20 at 12:17, Bill Mullen wrote: I suspect that the real issue here is merely one of storage space; by setting a fixed period for which they will make data available (last 90 days, last 3 statement periods, whatever), they can move enough transactions out of the database to keep up with the new transactions being added, all while keeping their online storage capacity fairly static and predictable. It's not that even -- online, a full 18 months of data is available; I am only allowed to access it 90 days at a time. The 90-day window can be *anywhere* in the total dataset; I just can't pull more than a 90-day chunk of it at any given time. The bank considers the data /their/ data, and is loathe to give you too much of it. If you could download all of it, it would make it too easy for you to export the data to another financial institution or self-manage your money. They don't want to lose the customer, or lessen the customers dependency on the bank's system(s). They'll let you have answers to specific balance and transaction questions (after all there is a lot of cost-savings in this type of self-service), up to the point of asking for 3 months worth of history at a time. But the second you ask a question like Can I have all my account history?, financial service companies only think about risk. E.g. Where are you transferring your account to? or, Is there a problem? (Are you thinking of suing us?) The other thing about historical data is that once you have it, it becomes easier to do a performance analysis. This is another taboo for financial services companies. The only performance numbers they want you to see are controlled numbers that they put out. One example of this is when a money management firm is doing poorly, they will quote the market performance in statement stuffers. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
On Fri, 2004-11-19 at 23:56, Fred wrote: So, maybe some of you have had some banking experience. Is there some sort of obscure Federal regulation or similar that stipulates that financial institutions can only allow a max of 90 days of data at a time to the customer? And why would their be such an inane restriction, anyway? I suspect that the real issue here is merely one of storage space; by setting a fixed period for which they will make data available (last 90 days, last 3 statement periods, whatever), they can move enough transactions out of the database to keep up with the new transactions being added, all while keeping their online storage capacity fairly static and predictable. If there's one thing that bankers like, it's static and predictable ... especially when it comes to expense items. ;) Years ago, before online banking existed, I was a mainframe operator for a statewide bank in Concord. We had the same policy then for the account history data that we made available to the tellers over the online system - current statement period plus the last two, that's it; everything beyond that was archived. Disk packs aren't cheap. :) -- Bill Mullen RLU# 270075 ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: OT -- 90-day limits in the financial world for downloading your data.
On Sat, 2004-11-20 at 12:17, Bill Mullen wrote: I suspect that the real issue here is merely one of storage space; by setting a fixed period for which they will make data available (last 90 days, last 3 statement periods, whatever), they can move enough transactions out of the database to keep up with the new transactions being added, all while keeping their online storage capacity fairly static and predictable. It's not that even -- online, a full 18 months of data is available; I am only allowed to access it 90 days at a time. The 90-day window can be *anywhere* in the total dataset; I just can't pull more than a 90-day chunk of it at any given time. If there's one thing that bankers like, it's static and predictable ... especially when it comes to expense items. ;) Sounds like a religion. Years ago, before online banking existed, I was a mainframe operator for a statewide bank in Concord. We had the same policy then for the account history data that we made available to the tellers over the online system - current statement period plus the last two, that's it; everything beyond that was archived. Disk packs aren't cheap. :) Yes, I recall disk packs with glee. I wrote a driver for one way back when, back in 1980. The drive itself was the size of a washing machine, and the disk pack itself was about as tall as, say, a typical monitor is today. And no, I did NOT make the drive dance across the floor. Our clients would not have been happy about that! :-) -- Fred -- [EMAIL PROTECTED] -- place [hey] in your subject. The mass of humans on planet Earth -- regard them as the ebbing seas in the winds of change. They ebb, they flow, they know not where to go. ___ gnhlug-discuss mailing list [EMAIL PROTECTED] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
