[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
Hi,
I am not sure I fully understand what your trying to do but maybe
something here will help. :)
for a bit.ly url like http://bit.ly/2FgTDX
To preview it you can add either a + to the end of it or insert /info/
http://bit.ly/2FgTDX+
http://bit.ly/info/2FgTDX
They also have an API...
http://code.google.com/p/bitly-api/wiki/ApiDocumentation
http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
So you could use a JSON
http://api.bit.ly/expand?version=2.0.1&shortUrl=http://bit.ly/2FgTDX&login=bitlyapidemo&apiKey=R_0da49e0a9118ff35f52f629d2d71bf07
Or JSONP (&callback=foo)
http://api.bit.ly/expand?version=2.0.1&shortUrl=http://bit.ly/2FgTDX&callback=foo&login=bitlyapidemo&apiKey=R_0da49e0a9118ff35f52f629d2d71bf07
or by Hash
http://api.bit.ly/info?version=2.0.1&hash=2FgTDX&login=bitlyapidemo&apiKey=R_0da49e0a9118ff35f52f629d2d71bf07
hope something there help :)
Cheers!
Vision Jinx
@visionjinx
On Jul 15, 7:14 am, Bjoern wrote:
> Hi,
>
> The problem is: how to find tweets that point to a certain URL. Most
> of those tweets would use a shortened version of the URL, and most of
> them probably bit.ly. Bit.ly does not provide a way to list all
> shortened versions of a URL, and creates individually shortened
> versions of each URL per User (namely, the twitter user, so there is
> no way to find the URL that is created via Twitter status updates -
> which would probably be the most common URL).
>
> I thought I had found a way to at least approximate searching for
> bit.ly-masked references on twitter with the /stats method from bitly.
> Giving a bitly hash, the /stats method lists referrers by urls, which
> might look like this (part of a /stats response):
>
> "twitter.com": { "/": 1, "/testFollowAPI": 1 } }
>
> So I figured that I could then take all the paths that are not common
> Twitter paths (like /home, /, /replies, /favorites, /inbox) and assume
> that they are twitter accounts.
>
> Then I searched Twitter for tweets from these accounts containing
> "bit.ly". For each hit, I would then again ask bit.ly to expand the
> bit.ly url in the tweet. With any luck, I would find the tweet
> referencing the url.
>
> Apart from the fact that it is a pretty wasteful process (lots of
> calls to bitly and also to Twitter search), I only just realized that
> of course /stats only lists references when somebody has clicked on a
> bit.ly URL. Since I don't expect many people to click on the URL on a
> user's profile page, the utility of this approach is greatly
> diminished. I think most of the referrers would just be /home if users
> click on tweets in their stream.
>
> I don't suppose Twitter would be willing to "click" on each incoming
> bit.ly link with referrer from the tweeting user's page at least
> once? ;-)
>
> I gave the long story because I want to emphasize that as far as I
> know, there still is no proper solution to this. Bit.ly does NOT
> provide a way to list all shortened versions of an URL.
>
> If there is such a method, please point me to it, because I just don't
> see it.
>
> I filed an issue for this a while ago, but it was closed with the
> announcement that it would be solved sometime in the future.
> Unfortunately there was not new "issue" or "feature" created that I
> could track to see when it would be solved.
>
> The original issue is
> here:http://code.google.com/p/twitter-api/issues/detail?id=402
>
> Sorry for the long post - I just wasted so much time trying to work
> around the issue, and only realized the flaws in the approach right
> now...
>
> Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 6:14 AM, Bjoern wrote: > > Hi, > > The problem is: how to find tweets that point to a certain URL. Most > of those tweets would use a shortened version of the URL, and most of > them probably bit.ly. Bit.ly does not provide a way to list all > shortened versions of a URL, and creates individually shortened > versions of each URL per User (namely, the twitter user, so there is > no way to find the URL that is created via Twitter status updates - > which would probably be the most common URL). The solution is to go about this the opposite way. Bit.ly will return the same shortened URL for any request using the same source URL. So, use bit.ly to shorten the URL, then search Twitter for the bit.ly URL you get back. The same is true for other URL shortening services and the more popular ones all have APIs. Nick
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 10:17 AM, Nick Arnett wrote: > > > On Wed, Jul 15, 2009 at 6:14 AM, Bjoern wrote: > >> >> Hi, >> >> The problem is: how to find tweets that point to a certain URL. Most >> of those tweets would use a shortened version of the URL, and most of >> them probably bit.ly. Bit.ly does not provide a way to list all >> shortened versions of a URL, and creates individually shortened >> versions of each URL per User (namely, the twitter user, so there is >> no way to find the URL that is created via Twitter status updates - >> which would probably be the most common URL). > > > The solution is to go about this the opposite way. > > Bit.ly will return the same shortened URL for any request using the same > source URL. So, use bit.ly to shorten the URL, then search Twitter for > the bit.ly URL you get back. The same is true for other URL shortening > services and the more popular ones all have APIs. > > Nick > But I believe bit.ly returns different, unique URLs for logged-in users, and some people/clients are autoshortening their own URLs in various fashions for various reasons. Thanks- - Andy Badera - and...@badera.us - Google me: http://www.google.com/search?q=andrew+badera - This email is: [ ] bloggable [x] ask first [ ] private
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 7:19 AM, Andrew Badera wrote: > >> > But I believe bit.ly returns different, unique URLs for logged-in users That is an option, but in my experience, it is relatively rare. Nick
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 4:27 pm, Nick Arnett wrote: > On Wed, Jul 15, 2009 at 7:19 AM, Andrew Badera wrote: > > > But I believe bit.ly returns different, unique URLs for logged-in users > > That is an option, but in my experience, it is relatively rare. If you want to create a bitly url via the API, you have to be logged in. Therefore I assume "twitter" logs in to bit.ly to create the URLs. Therefore I would expect all bit.ly URLs that Twitter generates to be specific to Twitter. If you shorten a URL via some other account (you always have to login for the bit.ly API), it would result in a different URL. If Twitter could generate the "standard URL", it would be a big step. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 4:17 pm, Nick Arnett wrote: > The solution is to go about this the opposite way. > > Bit.ly will return the same shortened URL for any request using the same > source URL. So, use bit.ly to shorten the URL, then search Twitter for the > bit.ly URL you get back. The same is true for other URL shortening services > and the more popular ones all have APIs. That is exactly what I was trying to do, and it worked reasonably well for tinyurl (since the option for individual URLs with tinyurl was not so prominent). However, with bit.ly and I think most other popular shorteners the default now are individual URLs. Therefore that approach does not help much anymore. Also, one would have to search Twitter for all possible shortened URLs - that is the next problem. For now I would be happy if I could cover bit.ly and perhaps four ot five other ones. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 4:04 pm, Vision Jinx wrote: > They also have an API... > > http://code.google.com/p/bitly-api/wiki/ApiDocumentation > > http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API Yes, but they don't offer a way to see all variations of a URL, so it does not help much. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 7:55 AM, Bjoern wrote: > > > > On Jul 15, 4:27 pm, Nick Arnett wrote: > > On Wed, Jul 15, 2009 at 7:19 AM, Andrew Badera wrote: > > > > > But I believe bit.ly returns different, unique URLs for logged-in > users > > > > That is an option, but in my experience, it is relatively rare. > > If you want to create a bitly url via the API, you have to be logged > in. Therefore I assume "twitter" logs in to bit.ly to create the URLs. > Therefore I would expect all bit.ly URLs that Twitter generates to be > specific to Twitter. If you shorten a URL via some other account (you > always have to login for the bit.ly API), it would result in a > different URL. If Twitter could generate the "standard URL", it would > be a big step. Hmmm... just tried a few and sure enough, the bit.ly URLs generated by Twitter seem to be unique to Twitter, although consistent. Apparently Twitter has enabled bit.ly URL tracking. There's a horrible solution to that, too... tweet the original URL and then read back the status to get the Twitter-specific bit.ly URL. Ugh. Nick
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 5:04 pm, Nick Arnett wrote: > There's a horrible solution to > that, too... tweet the original URL and then read back the status to get the > Twitter-specific bit.ly URL. Ugh. Actually that is a pretty good idea, thanks!!! It is horrible, but I can't think of a better way atm. I think twiturly have mentioned that they can resolve URLs to tweets, but their API is very limited. My current application is not cool enough to ask them for a higher rate limit ;-) Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 5:18 pm, Bjoern wrote: > Actually that is a pretty good idea, thanks!!! Argh, except that Twitter rate limits will bite me :-( What I have implemented is a search web site that shows associated tweets to the URLs, so potentially it would generate a lot of requests (one page of search results is 10 URLs to check). Better apply for whitelisting now... Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 11:21 am, Bjoern wrote: > Argh, except that Twitter rate limits will bite me :-( What I have > implemented is a search web site that shows associated tweets to the > URLs, so potentially it would generate a lot of requests (one page of > search results is 10 URLs to check). > > Better apply for whitelisting now... Not so fast... If I'm understanding you, the proposed solution is that for each non- shortened URL you want to search Twitter for, you send it in a status update, and then retrieve the shortened version by reading back that status, and then search Twitter for the shortened version. You can actually process 10 (or more) URLs with only one hit against your rate limit. How? Status updates don't count toward your rate limit (although Twitter may separately notice a large number of updates which are nothing more than URLs and mark you a spammer or something, but that's another discussion). So for 10 URLs, you post 10 status updates, then retrieve your own last 10 updates in one call by retrieving your own timeline via / statuses/user_timeline(and that's the one hit against your rate limit).
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 8:45 AM, Bill Kocik wrote: > > > So for 10 URLs, you post 10 status updates, then retrieve your own > last 10 updates in one call by retrieving your own timeline via / > statuses/user_timeline(and that's the one hit against your rate limit). If Twitter will shorten multiple URLs in the same tweet, you could get even more than that. I just tried putting two longer URLs in a tweet and it didn't shorten them at all, just did the ellipsis thing, so that was inconclusive. This method is rather unreliable, I suppose... and I don't want to post more test tweets. My mother will see them on Facebook and become confused. ;-) Nick
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 5:45 pm, Bill Kocik wrote: > Status updates don't count toward your rate limit > (although Twitter may separately notice a large number of updates > which are nothing more than URLs and mark you a spammer or something, > but that's another discussion). Interesting, thanks! I did not think of the different limits. For updates there seems to be a limit of 1000 per day: http://help.twitter.com/forums/10711/entries/15364 But it might be a start as long as my application is not very popular. Atm it is more a testbed for me - although I think in general the application of showing tweets referencing a URL is a valid interest. I think the only solutions in the wild use something like twitpic, where you enter a message at twitpic and twitpic ads an identifier to find the message (the url to twitpic in that case). But it would be kind of cool to be able to do that for everything, for example on blog posts. I have been wondering if for blogs it would be easier to analyze the referrer log, but I don't think it would work. Referrer would either be just bit.ly (?), or it would be twitter.com/ home and stuff like that. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Wed, Jul 15, 2009 at 11:04 AM, Nick Arnett wrote: > > Hmmm... just tried a few and sure enough, the bit.ly URLs generated by > Twitter seem to be unique to Twitter, although consistent. Apparently > Twitter has enabled bit.ly URL tracking. There's a horrible solution to > that, too... tweet the original URL and then read back the status to get the > Twitter-specific bit.ly URL. Ugh. > > Nick > Those generated BY Twitter, sure. But plenty of people are using clients or plugins that are doing their own shortening, whether by bit.ly or otherwise, so they can track their own stats. If I go tweet a bit.ly URL I created in my own personal account with bit.ly, it will stick with my bit.ly URL, not a Twitter-gen'd bit.ly URL.
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
Hi there, Please be aware there are update limits in addition to the rate limit. There are also the spam and abuse marshals looking out for accounts acting suspiciously. Posting a bunch of link-only tweets seems like it's very likely to run afoul of them and get the account suspended. I can't say for sure since they're always evolving the types of abuse the check for but I don't recommend this course of action. Have you thought about using one of the APIs built for this, like backtweets [1]? Thanks; – Matt Sanford / @mzsanford Twitter Dev [1] - Disclaimer: I've not used the API at http://backtweets.com/api but it seems like what you're looking for. On Jul 15, 2009, at 8:52 AM, Nick Arnett wrote: On Wed, Jul 15, 2009 at 8:45 AM, Bill Kocik wrote: So for 10 URLs, you post 10 status updates, then retrieve your own last 10 updates in one call by retrieving your own timeline via / statuses/user_timeline(and that's the one hit against your rate limit). If Twitter will shorten multiple URLs in the same tweet, you could get even more than that. I just tried putting two longer URLs in a tweet and it didn't shorten them at all, just did the ellipsis thing, so that was inconclusive. This method is rather unreliable, I suppose... and I don't want to post more test tweets. My mother will see them on Facebook and become confused. ;-) Nick
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 5:57 pm, Matt Sanford wrote: > Have you thought about using one of the APIs built for this, > like backtweets [1]? I thought about them, but only as a last resort. Did not know about backtweets - they look good, but they also have a limit of 1000 calls/ day. I had also looked into FriendFeed, but they seem to only return people who are also on FriendFeed. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
There are 3 API's that I know of that you can use: Twitturly (Ours - Private beta only at the moment) Tweetmeme BackTweet Between the 3 of us, I am sure you can accomplish whatever your end-goal is. I do not think BackTweet processes all URLs, so they may not have a URL, but I do know that we do and Tweetmeme does. -Joel -Original Message- From: twitter-development-talk@googlegroups.com [mailto:twitter-development-t...@googlegroups.com] On Behalf Of Bjoern Sent: Wednesday, July 15, 2009 9:16 AM To: Twitter Development Talk Subject: [twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others) On Jul 15, 5:57 pm, Matt Sanford wrote: > Have you thought about using one of the APIs built for this, > like backtweets [1]? I thought about them, but only as a last resort. Did not know about backtweets - they look good, but they also have a limit of 1000 calls/ day. I had also looked into FriendFeed, but they seem to only return people who are also on FriendFeed. Björn
[twitter-dev] Re: Searching for tweets that refer to an URL still impossible with bit.ly (and others)
On Jul 15, 6:36 pm, "Joel Strellner" wrote: > There are 3 API's that I know of that you can use: > > Twitturly (Ours - Private beta only at the moment) > Tweetmeme > BackTweet > > Between the 3 of us, I am sure you can accomplish whatever your end-goal is. Thanks - they are better than nothing, but they all have limitations. Not only rate limits, but for example I just read at Tweetmeme that they only keep URL references for the last 7 days. Polling as they suggest is not an option for me, as I don't know the URLs in advance. So I guess I'll have to wait for the Twiturly offering. Björn
