[google-appengine] Re: Twitter - rate limit exceeded
Thanks Tim. Think I've managed to convince myself that I can work around the lack of inbuilt scheduled tasks. Who knows by the time I manage to pull together enough motivation google may have implemented it already. Worst case I may be able to use a work server to call an URL, although its not a work project. Hmmm, wonder how that will go down. There's also http://schedulerservice.appspot.com/, which I might try. It's just the twitter rate limit roulette game I'm worried about now. Really highlights learnings from my first project, make sure you upload and test your app early. On Mar 16, 7:23 am, Tim Bull tim.b...@binaryplex.com wrote: Ahh! Yes, by google search API I meant Twitter search API! I'm using a CRON job to trigger a special URL every 5 minutes. Originally I had this job on my own webhost, but I breached the terms of service because a) sometimes the way I update the trend lists can take a long time and the very basic PHP fetch I do was waiting for a return value (which it doesn't really need to do) - this caused CPU limits on my cheap host to be exceeded and b) my cheap host only allows jobs to be scheduled every 15 minutes! I ended up with a two part solution: 1) I usehttp://www.webcron.orgto schedule jobs that call a URL on my webhost for longer jobs every 5 minutes or direct on GAE for shorter jobs. Webcron charges by the length of job so sub-30 seconds is cheapest (0.0001 Euro cents or 1000 jobs per cent) 2) On my webhost I use cURL instead of a standard PHP fetch (which is how I first did it) - this just triggers the job then terminates the script. GAE will happily continue to execute the job even though the listening party has terminated. I get what I want and my webhost doesn't get upset. I need to do it in this 2-part way becase webcron won't let you terminate a job after calling it - this achieved what I wanted in a fairly cheap way for me. Here is the PHP script I use Note the URL doesn't need the HTTP:// part in front of it. ? $url = myurl.appsot.com/somejob; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_TIMEOUT, 2); $curl_scraped_page = curl_exec($ch); curl_close($ch); ? On Sun, Mar 15, 2009 at 4:52 PM, lock lachlan.hu...@gmail.com wrote: Hi Tim, Just had a look at Twendly, looks good! I've just got a few quick questions, if you wouldn't mind... 1. By 'google search API' you actually mean 'twitter seach API', yeah ? ;-) 2. How do you go about pulling data from twitter every 5 minutes? Unless I'm missing something there are no scheduled tasks in app engine (yet). Using a cron job on another server to call a special URL maybe? The API key sounds like the proper solution, would be nice if there was a solution now though. Just an idea that probably won't work for most cases. Get the client (via javascript) to pull data from twitter and send it on to app engine for processing/storage. Not real pretty. Thanks, lock On Mar 15, 9:16 am, Tim Bull tim.b...@binaryplex.com wrote: Interesting, I have a Twitter app (http://twendly.appspot.com) but I don't seem to be having this issue at the moment. However, while I read information every 5 minutes from the google search API (which is rate limited differently) I only send a few messages (no more than 5 or 6 max and usually only 4) as the hour clicks over. Although ocasionally this drops a message, it's generally pretty solid. Perhaps because of when I'm sending them, I get in at the start of the allocation. As far as scalability goes, I would say GAE is really suited for it's read scalability, so if unless your Twitter bot writes are going to massive, then scalability shouldn't be an issue if you move these writes over to a seperate host. I guess a (nasty but possible) pattern would be to have the Twitter interaction come from your host which could act as a proxy, then use App Engine for all the processing and reporting on the data. At least in my application this would be a potential work-around if this becomes an issue. Cheers Tim --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Ahh! Yes, by google search API I meant Twitter search API! I'm using a CRON job to trigger a special URL every 5 minutes. Originally I had this job on my own webhost, but I breached the terms of service because a) sometimes the way I update the trend lists can take a long time and the very basic PHP fetch I do was waiting for a return value (which it doesn't really need to do) - this caused CPU limits on my cheap host to be exceeded and b) my cheap host only allows jobs to be scheduled every 15 minutes! I ended up with a two part solution: 1) I use http://www.webcron.org to schedule jobs that call a URL on my webhost for longer jobs every 5 minutes or direct on GAE for shorter jobs. Webcron charges by the length of job so sub-30 seconds is cheapest (0.0001 Euro cents or 1000 jobs per cent) 2) On my webhost I use cURL instead of a standard PHP fetch (which is how I first did it) - this just triggers the job then terminates the script. GAE will happily continue to execute the job even though the listening party has terminated. I get what I want and my webhost doesn't get upset. I need to do it in this 2-part way becase webcron won't let you terminate a job after calling it - this achieved what I wanted in a fairly cheap way for me. Here is the PHP script I use Note the URL doesn't need the HTTP:// part in front of it. ? $url = myurl.appsot.com/somejob; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_TIMEOUT, 2); $curl_scraped_page = curl_exec($ch); curl_close($ch); ? On Sun, Mar 15, 2009 at 4:52 PM, lock lachlan.hu...@gmail.com wrote: Hi Tim, Just had a look at Twendly, looks good! I've just got a few quick questions, if you wouldn't mind... 1. By 'google search API' you actually mean 'twitter seach API', yeah ? ;-) 2. How do you go about pulling data from twitter every 5 minutes? Unless I'm missing something there are no scheduled tasks in app engine (yet). Using a cron job on another server to call a special URL maybe? The API key sounds like the proper solution, would be nice if there was a solution now though. Just an idea that probably won't work for most cases. Get the client (via javascript) to pull data from twitter and send it on to app engine for processing/storage. Not real pretty. Thanks, lock On Mar 15, 9:16 am, Tim Bull tim.b...@binaryplex.com wrote: Interesting, I have a Twitter app (http://twendly.appspot.com) but I don't seem to be having this issue at the moment. However, while I read information every 5 minutes from the google search API (which is rate limited differently) I only send a few messages (no more than 5 or 6 max and usually only 4) as the hour clicks over. Although ocasionally this drops a message, it's generally pretty solid. Perhaps because of when I'm sending them, I get in at the start of the allocation. As far as scalability goes, I would say GAE is really suited for it's read scalability, so if unless your Twitter bot writes are going to massive, then scalability shouldn't be an issue if you move these writes over to a seperate host. I guess a (nasty but possible) pattern would be to have the Twitter interaction come from your host which could act as a proxy, then use App Engine for all the processing and reporting on the data. At least in my application this would be a potential work-around if this becomes an issue. Cheers Tim --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Interesting, I have a Twitter app (http://twendly.appspot.com) but I don't seem to be having this issue at the moment. However, while I read information every 5 minutes from the google search API (which is rate limited differently) I only send a few messages (no more than 5 or 6 max and usually only 4) as the hour clicks over. Although ocasionally this drops a message, it's generally pretty solid. Perhaps because of when I'm sending them, I get in at the start of the allocation. As far as scalability goes, I would say GAE is really suited for it's read scalability, so if unless your Twitter bot writes are going to massive, then scalability shouldn't be an issue if you move these writes over to a seperate host. I guess a (nasty but possible) pattern would be to have the Twitter interaction come from your host which could act as a proxy, then use App Engine for all the processing and reporting on the data. At least in my application this would be a potential work-around if this becomes an issue. Cheers Tim On Sat, Mar 14, 2009 at 3:57 PM, Richard Bremner richyr...@gmail.comwrote: Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
I could think of a rather simple solution for twitter: Just give everyone an API key that they use to authenticate, and make the limits based on the key instead of the IP addresses. No? On Mar 14, 12:57 am, Richard Bremner richyr...@gmail.com wrote: Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
That would work, and some Twitter API calls are authenticated already, but I guess it would be easy to bypass the limit by registering for multiple keys and having your app vary which key it uses. Maybe that doesn't matter in the big picture though. rich 2009/3/15 MajorProgamming sefira...@gmail.com I could think of a rather simple solution for twitter: Just give everyone an API key that they use to authenticate, and make the limits based on the key instead of the IP addresses. No? On Mar 14, 12:57 am, Richard Bremner richyr...@gmail.com wrote: Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Hi Tim, I presume GAE requests can come from a large number of different IP's. My app has been working fine for months, but then I made a small change, did an upload and it suddenly stopped working with this error, so I can only assume my app started coming from a different IP due to that upload, and the new IP had already made a lot of Twitter requests. Can the apps spontaneously change IP? I don't know enough about Google's infrastructure to diagnose. It could be that your app will be blocked the next time you upload it, that's what happened to me. Good luck with your upload roulette :-) rich 2009/3/15 Tim Bull tim.b...@binaryplex.com Interesting, I have a Twitter app (http://twendly.appspot.com) but I don't seem to be having this issue at the moment. However, while I read information every 5 minutes from the google search API (which is rate limited differently) I only send a few messages (no more than 5 or 6 max and usually only 4) as the hour clicks over. Although ocasionally this drops a message, it's generally pretty solid. Perhaps because of when I'm sending them, I get in at the start of the allocation. As far as scalability goes, I would say GAE is really suited for it's read scalability, so if unless your Twitter bot writes are going to massive, then scalability shouldn't be an issue if you move these writes over to a seperate host. I guess a (nasty but possible) pattern would be to have the Twitter interaction come from your host which could act as a proxy, then use App Engine for all the processing and reporting on the data. At least in my application this would be a potential work-around if this becomes an issue. Cheers Tim On Sat, Mar 14, 2009 at 3:57 PM, Richard Bremner richyr...@gmail.comwrote: Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Hi Tim, Just had a look at Twendly, looks good! I've just got a few quick questions, if you wouldn't mind... 1. By 'google search API' you actually mean 'twitter seach API', yeah ? ;-) 2. How do you go about pulling data from twitter every 5 minutes? Unless I'm missing something there are no scheduled tasks in app engine (yet). Using a cron job on another server to call a special URL maybe? The API key sounds like the proper solution, would be nice if there was a solution now though. Just an idea that probably won't work for most cases. Get the client (via javascript) to pull data from twitter and send it on to app engine for processing/storage. Not real pretty. Thanks, lock On Mar 15, 9:16 am, Tim Bull tim.b...@binaryplex.com wrote: Interesting, I have a Twitter app (http://twendly.appspot.com) but I don't seem to be having this issue at the moment. However, while I read information every 5 minutes from the google search API (which is rate limited differently) I only send a few messages (no more than 5 or 6 max and usually only 4) as the hour clicks over. Although ocasionally this drops a message, it's generally pretty solid. Perhaps because of when I'm sending them, I get in at the start of the allocation. As far as scalability goes, I would say GAE is really suited for it's read scalability, so if unless your Twitter bot writes are going to massive, then scalability shouldn't be an issue if you move these writes over to a seperate host. I guess a (nasty but possible) pattern would be to have the Twitter interaction come from your host which could act as a proxy, then use App Engine for all the processing and reporting on the data. At least in my application this would be a potential work-around if this becomes an issue. Cheers Tim On Sat, Mar 14, 2009 at 3:57 PM, Richard Bremner richyr...@gmail.comwrote: Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
You could host one script on an other server that works as a proxy for your requests to twitter. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Twitter - rate limit exceeded
Hmmm yes this is a difficult one. Neither Twitter nor Google are being unreasonable, and each GAE developer is probably performing a sane number of Twitter API requests but combined we are ruining it for everyone. Ohhh the solution? I can't think of a good solution Twitter could implement which wouldn't make it easy to circumvent their limit unreasonably. I do happen to have a hosted linux server a I can put a proxy script on, I guess I'm lucky there, but I am using GAE for its scaleability which my server certainly isn't. I don't need to go into all the reasons GAE is more scaleable than my own server :-) If anyone thinks of anything, I'd love to know. Rich 2009/3/14 lock lachlan.hu...@gmail.com Hmmm. My next app engine project _was_ going to be an app that relied on twitter. This doesn't sound good. As per your situation the app wouldn't hammer twitter, one request to the search API every 5-10 minutes or so. Given its not exactly an app engine problem did you try contacting twitter to see if they could build more 'smarts' into their rate limiting? Would be really interested to see if you end up resolving this issue, thanks for the heads up. Sorry I can't help. Cheers, lock On Mar 12, 10:43 pm, richyrich richyr...@gmail.com wrote: Hi there, I have been writing a simple little app that uses the Twitter API. It works perfectly on my local development server but it fails when I upload it because I get this error from Twitter: error=Rate limit exceeded. Clients may not make more than 100 requests per hour. ...even though my app only makes 1 request. what is happening is that other people apps must be using the Twitter API from the same IP address. does anyone know a good way around this other than hosting my app somewhere else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
