I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data. I'd sign agreements and pay money for such.
dave On Jul 28, 12:03 pm, owkaye <owk...@gmail.com> wrote: > I agree with you Dave. I have several thought about new > services based on searching Twitter's historical data. > Unfortunately my ideas appear to be getting less and less > practical. > > Twitter claims to have all its data stored in disk-based > databases from what I understand ... yet without access to > this data it is worthless. > > It seems to me they could allow searches of this historical > data via a new "History API" then let us cache the results > on our own servers. Most of the services I've conceived > would do this infrequently -- never in real time -- and > would not impact their existing "cached" server data because > this historical data would exist on separate data storage > servers ... theoretically anyways. > > Owkaye > > > > > I am a bit concerned. I remember at one point it being > > between 30-45 days. Now it seems to be getting smaller by > > about 1-day per month. Last month it was closer to 10 > > days. > > > Is it basically going to keep getting smaller and smaller > > until we get V2 of the API, or will we be forced to all > > use only streaming services and then locally cache > > everything that we'd want to search for any time period? > > > I know there are a LOT of problems inherent in the > > massive scaling out of Twitter, and this is just a > > symptom of them- but at the same time I can only imagine > > how unusable Google would be if you only had a 7-day > > window to Search in, and couldn't get any content made > > prior to that. Very worried about this soon being a 2-3 > > day window. > > > dave
