This is happening RIGHT NOW for the following: 1) Go to search.twitter.com and enter "tweetsforboobs OR tweetforboobs" as the search.
2) Go to http://tweetsforboobs.org and see the twitter feed on the left. Notice that the last tweet from 2 hours ago (VerticalMeasures) is not in the twitter feed on tweetsforboobs.org. Also note the ID of the tweet - from VerticalMeasures that is missing from tweetsforboobs.org: 5181937429 Now here is the log file of the Twitter API call: DEBUG: 06:18:01 PM on Mon October 26th Doing CURL fetch with User Agent: justsignal/1.0 (+http://justsignal.com) and RFERER: http://justsignal.com/widgets/20ab5e90bf116397d6fb84ca80321928/widget.html DEBUG: 06:18:01 PM on Mon October 26th Twitter responded with 200 HTTP Status Code. DEBUG: 06:18:01 PM on Mon October 26th MaxID: 5182676703 DEBUG: 06:18:01 PM on Mon October 26th There are: 0 results in this fetch. Updating number for api hits for hour: 18 to: 6 THROTTLE-69: 06:18:01 PM on Mon October 26th Slowing collection... Avg: 0 returning delay: 180 DEBUG: 06:18:01 PM on Mon October 26th Checking for next page... **** DEBUG: 06:18:01 PM on Mon October 26th There is NOT another page of results... DEBUG: 06:18:01 PM on Mon October 26th Old max: 5182676703 New max: 5182676703 DEBUG: 06:18:01 PM on Mon October 26th Old max: 5182676703 New max: 5182676703 Note that our id is already > the last tweet ID from VerticalMeasures, yet we never got that tweet. Our id from the log snip: (5182676703) is NOT in our database (we never got it). It does not match the tweet ID before Vertical Measures: 5180513610 Somehow the API is returning a new (and bigger) max id on 200 responses with no tweets in them OR on 403 (those are the only two http codes in the log for today). Either way, that shouldn't be happening. Brian Roy justSignal On Oct 26, 12:47 pm, briantroy <[email protected]> wrote: > Everything below ONLY PERTAINS TO THE SEARCH API: > > 1) Since late last week I've noticed a significant number of 403 > errors (403 Error from JSON: since_id too recent, poll less > frequently). These usually indicate I'm hitting a server with an > "older" view of the search index - since it thinks the ID I sent in > since_id is newer than the newest it has. These trouble me because > when I get a 200 after the 403 sometimes I get everything back to my > since_id, sometimes I don't. I appears some indexes have gaps until > they catch up. > > QUESTION: Are there any ongoing search indexing issues that you are > aware of? > > 2) Since late last week I've noticed that some search API requests > appear to get "stuck" returning an empty json result (no new tweets). > This can go on for HOURS (today one got stuck like this for 12 hours). > When I restart my process sometimes this clears up (I get the backlog) > - other times it does not (I continue to get 0 tweets in the json). > All of the requests return HTTP 200 and valid json. > > QUESTION: Are they any ongoing caching issues with the search API? > > These issues are new in the last 7 days (since about last Thursday). > My IP is whitelisted. I'm sending both a valid user agent and referrer > header. My processes are throttled by the volume of tweets the > receive. I've made no changes to my processing since late September. > > Any assistance would be appreciated. My user's are comparing what they > see from my service to search.twitter.com and telling me we are > broken. > > Regards, > > Brian Roy > justSignal
