Re: [twitter-dev] Fetching a user's entire timeline

[Adriaan Pelzer]

Thanks all - I managed to get this fixed.

The main thing to remember is to build in some kind of delay between
successive API calls. In my case I made it 200ms with linear backoff on
retries. I know this is documented for the streaming api's - wasn't aware
that it also holds for the REST API.

Thanks Tom for your help - greatly appreciated! I'm getting the full 3200
tweets now. The thing to remember here is, include_rts doesn't default to
true, so if you don't set it, retweets are omitted, but still counted. This
is documented, I just didn't rtfm properly ...

Adriaan Pelzer

putting you in touch with your crowds
http://www.wewillraakyou.com
twitter: http://www.twitter.com/adriaan_pelzer
linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
skype: adriaan_pelzer

+4478 7978 1743



On Mon, Jul 4, 2011 at 11:44 AM, Adriaan Pelzer
wrote:

> Ah - thanks. That makes sense.
>
> Still, having to retry on empty results doesn't make sense, or is this
> normal behaviour when not using pagination?
>
> Adriaan Pelzer
>
> putting you in touch with your crowds
> http://www.wewillraakyou.com
> twitter:
> http://www.twitter.com/adriaan_pelzer
> linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
> skype: adriaan_pelzer
> 
> +4478 7978 1743
>
>
>
> On Mon, Jul 4, 2011 at 11:37 AM, Tom van der Woerdt  wrote:
>
>>  Check for whether you include retweets.
>>
>> Tom
>>
>>
>>
>> On 7/4/11 12:34 PM, Adriaan Pelzer wrote:
>>
>> I'm not paging, but rather using a combination of max_id and count=200.
>> Also - not getting nearly 200 every time, mostly below 100
>>
>> Adriaan Pelzer
>>
>>  putting you in touch with your crowds
>> http://www.wewillraakyou.com
>> twitter: http://www.twitter.com/adriaan_pelzer
>> linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
>> skype: adriaan_pelzer
>>  +4478 7978 1743
>>
>>
>>
>> On Mon, Jul 4, 2011 at 11:18 AM, Correa Denzil  wrote:
>>
>>> I am unable to reproduce the following error. I am able to paginate 200
>>> tweets for 16 times.
>>>
>>>  I am using OAuth for authentication.
>>>
>>> --Regards,
>>> Denzil
>>>
>>>
>>>
>>>
>>> On Mon, Jul 4, 2011 at 2:43 PM, Adriaan Pelzer <
>>> adri...@wewillraakyou.com> wrote:
>>>
 We realise that we're not supposed to get 3200 (but 3200 minus spammed
 tweets), but we're only getting at most 2000 tweets. We don't want to retry
 too aggressively, it eats into the rate limits, and we're concerned the
 account might be flagged as abusive.

>>>
>>>   --
>>> Twitter developer documentation and resources:
>>> https://dev.twitter.com/doc
>>> API updates via Twitter: https://twitter.com/twitterapi
>>> Issues/Enhancements Tracker:
>>> https://code.google.com/p/twitter-api/issues/list
>>> Change your membership to this group:
>>> https://groups.google.com/forum/#!forum/twitter-development-talk
>>>
>>
>>  --
>> Twitter developer documentation and resources:
>> https://dev.twitter.com/doc
>> API updates via Twitter: https://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> https://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> https://groups.google.com/forum/#!forum/twitter-development-talk
>>
>>
>>  --
>> Twitter developer documentation and resources:
>> https://dev.twitter.com/doc
>> API updates via Twitter: https://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> https://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> https://groups.google.com/forum/#!forum/twitter-development-talk
>>
>
>

-- 
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

Re: [twitter-dev] Fetching a user's entire timeline

[Adriaan Pelzer]

Ah - thanks. That makes sense.

Still, having to retry on empty results doesn't make sense, or is this
normal behaviour when not using pagination?

Adriaan Pelzer

putting you in touch with your crowds
http://www.wewillraakyou.com
twitter: http://www.twitter.com/adriaan_pelzer
linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
skype: adriaan_pelzer

+4478 7978 1743



On Mon, Jul 4, 2011 at 11:37 AM, Tom van der Woerdt  wrote:

>  Check for whether you include retweets.
>
> Tom
>
>
>
> On 7/4/11 12:34 PM, Adriaan Pelzer wrote:
>
> I'm not paging, but rather using a combination of max_id and count=200.
> Also - not getting nearly 200 every time, mostly below 100
>
> Adriaan Pelzer
>
>  putting you in touch with your crowds
> http://www.wewillraakyou.com
> twitter: http://www.twitter.com/adriaan_pelzer
> linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
> skype: adriaan_pelzer
>  +4478 7978 1743
>
>
>
> On Mon, Jul 4, 2011 at 11:18 AM, Correa Denzil  wrote:
>
>> I am unable to reproduce the following error. I am able to paginate 200
>> tweets for 16 times.
>>
>>  I am using OAuth for authentication.
>>
>> --Regards,
>> Denzil
>>
>>
>>
>>
>> On Mon, Jul 4, 2011 at 2:43 PM, Adriaan Pelzer > > wrote:
>>
>>> We realise that we're not supposed to get 3200 (but 3200 minus spammed
>>> tweets), but we're only getting at most 2000 tweets. We don't want to retry
>>> too aggressively, it eats into the rate limits, and we're concerned the
>>> account might be flagged as abusive.
>>>
>>
>>   --
>> Twitter developer documentation and resources:
>> https://dev.twitter.com/doc
>> API updates via Twitter: https://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> https://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> https://groups.google.com/forum/#!forum/twitter-development-talk
>>
>
>  --
> Twitter developer documentation and resources: https://dev.twitter.com/doc
> API updates via Twitter: https://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> https://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> https://groups.google.com/forum/#!forum/twitter-development-talk
>
>
>  --
> Twitter developer documentation and resources: https://dev.twitter.com/doc
> API updates via Twitter: https://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> https://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> https://groups.google.com/forum/#!forum/twitter-development-talk
>

-- 
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

Re: [twitter-dev] Fetching a user's entire timeline

[Tom van der Woerdt]

Check for whether you include retweets.

Tom


On 7/4/11 12:34 PM, Adriaan Pelzer wrote:

I'm not paging, but rather using a combination of max_id and count=200.
Also - not getting nearly 200 every time, mostly below 100

Adriaan Pelzer

putting you in touch with your crowds
http://www.wewillraakyou.com
twitter: http://www.twitter.com/adriaan_pelzer
linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
skype: adriaan_pelzer
+4478 7978 1743



On Mon, Jul 4, 2011 at 11:18 AM, Correa Denzil > wrote:


I am unable to reproduce the following error. I am able to
paginate 200 tweets for 16 times.

I am using OAuth for authentication.

--Regards,
Denzil




On Mon, Jul 4, 2011 at 2:43 PM, Adriaan Pelzer
mailto:adri...@wewillraakyou.com>> wrote:

We realise that we're not supposed to get 3200 (but 3200 minus
spammed tweets), but we're only getting at most 2000 tweets.
We don't want to retry too aggressively, it eats into the rate
limits, and we're concerned the account might be flagged as
abusive.


-- 
Twitter developer documentation and resources:

https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker:
https://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
https://groups.google.com/forum/#!forum/twitter-development-talk



--
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: 
https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk 



--
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

Re: [twitter-dev] Fetching a user's entire timeline

[Adriaan Pelzer]

I'm not paging, but rather using a combination of max_id and count=200.
Also - not getting nearly 200 every time, mostly below 100

Adriaan Pelzer

putting you in touch with your crowds
http://www.wewillraakyou.com
twitter: http://www.twitter.com/adriaan_pelzer
linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
skype: adriaan_pelzer

+4478 7978 1743



On Mon, Jul 4, 2011 at 11:18 AM, Correa Denzil  wrote:

> I am unable to reproduce the following error. I am able to paginate 200
> tweets for 16 times.
>
> I am using OAuth for authentication.
>
> --Regards,
> Denzil
>
>
>
>
> On Mon, Jul 4, 2011 at 2:43 PM, Adriaan Pelzer 
> wrote:
>
>> We realise that we're not supposed to get 3200 (but 3200 minus spammed
>> tweets), but we're only getting at most 2000 tweets. We don't want to retry
>> too aggressively, it eats into the rate limits, and we're concerned the
>> account might be flagged as abusive.
>>
>
>  --
> Twitter developer documentation and resources: https://dev.twitter.com/doc
> API updates via Twitter: https://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> https://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> https://groups.google.com/forum/#!forum/twitter-development-talk
>

-- 
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

Re: [twitter-dev] Fetching a user's entire timeline

[Correa Denzil]

I am unable to reproduce the following error. I am able to paginate 200
tweets for 16 times.

I am using OAuth for authentication.

--Regards,
Denzil



On Mon, Jul 4, 2011 at 2:43 PM, Adriaan Pelzer wrote:

> We realise that we're not supposed to get 3200 (but 3200 minus spammed
> tweets), but we're only getting at most 2000 tweets. We don't want to retry
> too aggressively, it eats into the rate limits, and we're concerned the
> account might be flagged as abusive.
>

-- 
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

Re: [twitter-dev] Fetching a user's entire timeline

[Adriaan Pelzer]

We are noticing the exact same erratic behaviour. We also don't seem to
reach 3200 ever.

We realise that we're not supposed to get 3200 (but 3200 minus spammed
tweets), but we're only getting at most 2000 tweets. We don't want to retry
too aggressively, it eats into the rate limits, and we're concerned the
account might be flagged as abusive.

Could someone from Twitter give some guidelines as to what we can expect and
best practise?

Adriaan Pelzer

putting you in touch with your crowds
http://www.wewillraakyou.com
twitter: http://www.twitter.com/adriaan_pelzer
linkedIn: http://uk.linkedin.com/pub/adriaan-pelzer/4/874/860/
skype: adriaan_pelzer

+4478 7978 1743



On Fri, Feb 25, 2011 at 5:46 PM, James Wheare  wrote:

> OK so I'm not complaining about the 3200 limit here, but I would like
> to know the most robust way to fetch all 3200 of a user's available
> tweets.
>
> At the moment I'm calling user_timeline recursively, 100 tweets at a
> time (the maximum of 200 seems to timeout too often), passing in the
> oldest status id from one call onto the next as the max_id parameter.
>
> This usually works pretty well, but I'm relying on an assumption that
> I now realise doesn't always seem to hold: I'm expecting the response
> to only be an empty list when I reach the ~3200 limit. I've noticed
> with some timelines that an empty list will sporadically be returned
> well before this limit. If I keep trying the call I eventually get
> results, and for some accounts it happens quite regularly, often
> taking several attempts to get a response, but the empty list state
> triggers my script's "importing complete" end state prematurely.
> Obviously I can't rely on this assumption anymore.
>
> I'm handling errors fine, but this isn't an error, it's an HTTP 200
> response with an empty result set.
>
> I've seen other apps first check the user's tweet count, then divide
> Max(3200, total_count) by the number of tweets per page to get a page
> count, and simply page++ their way through user_timeline. This seems
> even less robust though, as the tweet count may sometimes change by
> the time you get to the end of the list, and also, I've noticed the
> 3200 limit isn't exactly a hard cutoff.
>
> Has anyone had any experience of this and can offer advice on how best
> to deal with it. Seems like the 200 status is a bug in the API too.
>
> Cheers,
> - James
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>

-- 
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
https://groups.google.com/forum/#!forum/twitter-development-talk

[twitter-dev] Fetching a user's entire timeline

[James Wheare]

OK so I'm not complaining about the 3200 limit here, but I would like
to know the most robust way to fetch all 3200 of a user's available
tweets.

At the moment I'm calling user_timeline recursively, 100 tweets at a
time (the maximum of 200 seems to timeout too often), passing in the
oldest status id from one call onto the next as the max_id parameter.

This usually works pretty well, but I'm relying on an assumption that
I now realise doesn't always seem to hold: I'm expecting the response
to only be an empty list when I reach the ~3200 limit. I've noticed
with some timelines that an empty list will sporadically be returned
well before this limit. If I keep trying the call I eventually get
results, and for some accounts it happens quite regularly, often
taking several attempts to get a response, but the empty list state
triggers my script's "importing complete" end state prematurely.
Obviously I can't rely on this assumption anymore.

I'm handling errors fine, but this isn't an error, it's an HTTP 200
response with an empty result set.

I've seen other apps first check the user's tweet count, then divide
Max(3200, total_count) by the number of tweets per page to get a page
count, and simply page++ their way through user_timeline. This seems
even less robust though, as the tweet count may sometimes change by
the time you get to the end of the list, and also, I've noticed the
3200 limit isn't exactly a hard cutoff.

Has anyone had any experience of this and can offer advice on how best
to deal with it. Seems like the 200 status is a bug in the API too.

Cheers,
- James

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk