Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Quoting DeWitt Clinton dclin...@gmail.com: Awesome, thanks for the quick response! Those are the right answers, too. : ) Though there's an inconsistency with returning 301's and also requiring every click to go through the t.co link (as required by the ToS). A 301 means that the redirect is cacheable by any intermediary (because it is permanent and will never change). The 301 also implies that you actually *can* replace only the malicious links, (not every link), because clients will already have resolved and cached the 301 redirects (which again, can never change), so you won't be able to change the redirect down the road anyway. So, I think you might actually have meant to use 302's, not 301's, if redirecting every click is the goal. But then again, 301's really are the (philosophically? morally?) right answer, so maybe I *don't* want you to fix that. : ) Or better still, resolving *all* URLs upfront and returning the full URL inline, making tweets longer than 140 characters, and stopping this whole URL shortening nonsense to begin with. (But you knew I'd say that...!) -DeWitt Well ... while we're on the subject ... since we're talking *tweets* and not arbitrary text from just anywhere, why do we have to waste seven characters with the http://;? Can't it just be t.co/abcdef? After all, the receiver got the message from Twitter, not from thin air, and can easily supply the http://;. You're never going to have an ftp://t.co;, file://t.co, etc., right?
[twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Hi Raffi,
Interesting... A couple of quick questions:
*1)* Will the redirect from t.co - domain.com be a 301 Moved Permanently or
a 302 Found response?
*2)* Will the t.co URL redirect point to the URL in the original tweet, or
will it point to the ultimate resolved URL?
I.e., if I post Check out my site at http://bit.ly/abcd; where
bit.ly/abcdredirects to
domain.com, and the resultant tweet becomes Check out my site at
http://t.co/abcd;, will the t.co URL redirect like this:
a) t.co/abcd - domain.com
Or like this:
b) t.co/abcd - bit.ly/abcd - domain.com
*3)* In the above scenario, will the 'display_url' contain '
http://bit.ly/abcd' or 'http://domain.com'?
*4)* Why redirect all URLs, btw? Why not just redirect the malicious ones?
Thanks!
-DeWitt
On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian ra...@twitter.com wrote:
hi all.
twitter has been wrapping links in e-mailed DMs for a couple months
nowhttp://bit.ly/twttldmemail.
with that feature, we're trying to protect users against phishing and other
malicious attacks. the way that we're doing this is that any URL that comes
through in a DM gets currently wrapped with a twt.tl URL -- if the URL
turns out to be malicious, Twitter can simply shut it down, and whoever
follows that link will be presented with a page that warns them of
potentially malicious content. in a few weeks, we're going to start slowly
enabling this throughout the API for all statuses as well, but instead of
twt.tl, we will be using t.co.
practically, any tweet that is sent through statuses/update that has a
link on it will have the link automatically converted to a t.co link on
its way through the Twitter platform. if you fetch any tweet created after
this change goes live, then its text field will have all its links
automatically wrapped with t.co links. when a user clicks on that link,
Twitter will redirect them to the original URL after first confirming with
our database that that URL is not malicious. on top of the end-user
benefit, we hope to eventually provide all developers with aggregate usage
data around your applications such as the number of clicks people make on
URLs you display (it will, of course, be in aggregate and not identifiable
manner). additionally, we want to be able to build services and APIs that
can make algorithmic recommendations to users based on the content they are
consuming. gathering the data from t.co will help make these possible.
our current plan is that no user will see a t.co URL on twitter.com but we
still have some details to work through. the links will still be displayed
as they were sent in, but the target of the link will be the t.co link
instead. and, we want to provide the same ability to display original links
to developers. we're going to use the entities attribute to make this
possible.
let's say i send out the following tweet: you have to check out
http://dev.twitter.com!;
a returned (and truncated) status object may look like:
{
text : you have to check out http://t.co/s9gfk2d4!;,
...
user : {
screen_name : raffi,
...
},
...
entities : {
urls : [
{
url : http://t.co/s9gfk2d4;,
display_url : http://dev.twitter.com;,
indices : [23, 43]
}
],
...
},
...
}
two things to note: the text of the returned status object doesn't have the
original URL and instead it has a t.co URL, and the entities block now has
a display_url attribute associated with it. what we're hoping is that with
this data, it should be relatively easy to create a UI where you replace the
http://t.co/s9gfk2d4 in the text with the equivalent of
a href=http://t.co/s9gfk2d4;http://dev.twitter.com/a
this means the user would not see the t.co link, but we all can still
provide the protection and gather data from the wrapped link. for the
applications that don't choose to update, the t.co link will be shown (and
the goal to protect users will be met). i just want to emphasize -- we
really do hope that you all render the original URL, but please send the
user through the t.co link. if you do choose to prefetch all the URLs on
a timeline, then, when a user actually clicks on one of the links, please
still send him or her through t.co. We will be updating the TOS to require
you to check t.co and register the click.
related to this: the way the Twitter API counts characters is going to
change ever so slightly. our 140 characters is now going to be defined as
140 characters after link wrapping. t.co links are of a predictable length
-- they will always be 20 characters. after we make this live, it will be
feasible to send in the text for a status that is greater than 140
characters. the rule is after the link wrapping, the text transforms to 140
characters or fewer. we'll be using the same logic that is in
twitter-text-rb to figure out what is a URL.
look for an update to dev.twitter.com where we'll have a
[twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Hi Raffi, On 9/06/10 8:57 AM, Raffi Krikorian wrote: url : http://t.co/s9gfk2d4;, display_url : http://dev.twitter.com;, indices : [23, 43] Any chance of getting the title of the resolved URL added in here too if available? Then we could display a link like : a title=Twitter Dev href=http://t.co/s9gfk2d4; http://dev.twitter.com /a or : a title=http://dev.twitter.com; href=http://t.co/s9gfk2d4; Twitter Dev /a This would give even more context to users, without having to follow the redirect path, load and parse the page to extract it as well. Thanks, JB.
Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API
*1)* Will the redirect from t.co - domain.com be a 301 Moved Permanently
or a 302 Found response?
301!
*2)* Will the t.co URL redirect point to the URL in the original tweet, or
will it point to the ultimate resolved URL?
I.e., if I post Check out my site at http://bit.ly/abcd; where
bit.ly/abcd redirects to domain.com, and the resultant tweet becomes
Check out my site at http://t.co/abcd;, will the t.co URL redirect like
this:
a) t.co/abcd - domain.com
Or like this:
b) t.co/abcd - bit.ly/abcd - domain.com
we're not modifying or tampering with URLs - if you send us a bit.ly link,
we will wrap that bit.ly link. analytics will still work, etc.
*3)* In the above scenario, will the 'display_url' contain '
http://bit.ly/abcd' or 'http://domain.com'?
bit.ly!
*4)* Why redirect all URLs, btw? Why not just redirect the malicious
ones?
in the case of malicious URLs, you sometimes don't know it at the time of
tweet creation. or the URL may eventually become malicious. this allows us
to do shutdown after tweet creation.
Thanks!
that's what i'm here for :P
-DeWitt
On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian ra...@twitter.com wrote:
hi all.
twitter has been wrapping links in e-mailed DMs for a couple months
nowhttp://bit.ly/twttldmemail.
with that feature, we're trying to protect users against phishing and other
malicious attacks. the way that we're doing this is that any URL that comes
through in a DM gets currently wrapped with a twt.tl URL -- if the URL
turns out to be malicious, Twitter can simply shut it down, and whoever
follows that link will be presented with a page that warns them of
potentially malicious content. in a few weeks, we're going to start slowly
enabling this throughout the API for all statuses as well, but instead of
twt.tl, we will be using t.co.
practically, any tweet that is sent through statuses/update that has a
link on it will have the link automatically converted to a t.co link on
its way through the Twitter platform. if you fetch any tweet created after
this change goes live, then its text field will have all its links
automatically wrapped with t.co links. when a user clicks on that link,
Twitter will redirect them to the original URL after first confirming with
our database that that URL is not malicious. on top of the end-user
benefit, we hope to eventually provide all developers with aggregate usage
data around your applications such as the number of clicks people make on
URLs you display (it will, of course, be in aggregate and not identifiable
manner). additionally, we want to be able to build services and APIs that
can make algorithmic recommendations to users based on the content they are
consuming. gathering the data from t.co will help make these possible.
our current plan is that no user will see a t.co URL on twitter.com but
we still have some details to work through. the links will still be
displayed as they were sent in, but the target of the link will be the
t.co link instead. and, we want to provide the same ability to display
original links to developers. we're going to use the entities attribute to
make this possible.
let's say i send out the following tweet: you have to check out
http://dev.twitter.com!;
a returned (and truncated) status object may look like:
{
text : you have to check out http://t.co/s9gfk2d4!;,
...
user : {
screen_name : raffi,
...
},
...
entities : {
urls : [
{
url : http://t.co/s9gfk2d4;,
display_url : http://dev.twitter.com;,
indices : [23, 43]
}
],
...
},
...
}
two things to note: the text of the returned status object doesn't have
the original URL and instead it has a t.co URL, and the entities block
now has a display_url attribute associated with it. what we're hoping is
that with this data, it should be relatively easy to create a UI where you
replace the http://t.co/s9gfk2d4 in the text with the equivalent of
a href=http://t.co/s9gfk2d4;http://dev.twitter.com/a
this means the user would not see the t.co link, but we all can still
provide the protection and gather data from the wrapped link. for the
applications that don't choose to update, the t.co link will be shown
(and the goal to protect users will be met). i just want to emphasize -- we
really do hope that you all render the original URL, but please send the
user through the t.co link. if you do choose to prefetch all the URLs
on a timeline, then, when a user actually clicks on one of the links, please
still send him or her through t.co. We will be updating the TOS to
require you to check t.co and register the click.
related to this: the way the Twitter API counts characters is going to
change ever so slightly. our 140 characters is now going to be defined as
140 characters after link wrapping. t.co links are of a predictable
length -- they will always be 20 characters.
Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API
that would be an awesome service! On Tue, Jun 8, 2010 at 4:50 PM, John Barratt djo...@gmail.com wrote: Hi Raffi, On 9/06/10 8:57 AM, Raffi Krikorian wrote: url : http://t.co/s9gfk2d4;, display_url : http://dev.twitter.com;, indices : [23, 43] Any chance of getting the title of the resolved URL added in here too if available? Then we could display a link like : a title=Twitter Dev href=http://t.co/s9gfk2d4; http://dev.twitter.com /a or : a title=http://dev.twitter.com; href=http://t.co/s9gfk2d4; Twitter Dev /a This would give even more context to users, without having to follow the redirect path, load and parse the page to extract it as well. Thanks, JB. -- Raffi Krikorian Twitter Platform Team http://twitter.com/raffi
Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Awesome, thanks for the quick response!
Those are the right answers, too. : )
Though there's an inconsistency with returning 301's and also requiring
every click to go through the t.co link (as required by the ToS). A 301
means that the redirect is cacheable by any intermediary (because it is
permanent and will never change).
The 301 also implies that you actually *can* replace only the malicious
links, (not every link), because clients will already have resolved and
cached the 301 redirects (which again, can never change), so you won't be
able to change the redirect down the road anyway.
So, I think you might actually have meant to use 302's, not 301's, if
redirecting every click is the goal.
But then again, 301's really are the (philosophically? morally?) right
answer, so maybe I *don't* want you to fix that. : )
Or better still, resolving *all* URLs upfront and returning the full URL
inline, making tweets longer than 140 characters, and stopping this whole
URL shortening nonsense to begin with. (But you knew I'd say that...!)
-DeWitt
On Tue, Jun 8, 2010 at 4:54 PM, Raffi Krikorian ra...@twitter.com wrote:
*1)* Will the redirect from t.co - domain.com be a 301 Moved Permanently
or a 302 Found response?
301!
*2)* Will the t.co URL redirect point to the URL in the original tweet,
or will it point to the ultimate resolved URL?
I.e., if I post Check out my site at http://bit.ly/abcd; where
bit.ly/abcd redirects to domain.com, and the resultant tweet becomes
Check out my site at http://t.co/abcd;, will the t.co URL redirect like
this:
a) t.co/abcd - domain.com
Or like this:
b) t.co/abcd - bit.ly/abcd - domain.com
we're not modifying or tampering with URLs - if you send us a bit.ly link,
we will wrap that bit.ly link. analytics will still work, etc.
*3)* In the above scenario, will the 'display_url' contain '
http://bit.ly/abcd' or 'http://domain.com'?
bit.ly!
*4)* Why redirect all URLs, btw? Why not just redirect the malicious
ones?
in the case of malicious URLs, you sometimes don't know it at the time of
tweet creation. or the URL may eventually become malicious. this allows us
to do shutdown after tweet creation.
Thanks!
that's what i'm here for :P
-DeWitt
On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian ra...@twitter.comwrote:
hi all.
twitter has been wrapping links in e-mailed DMs for a couple months
nowhttp://bit.ly/twttldmemail.
with that feature, we're trying to protect users against phishing and other
malicious attacks. the way that we're doing this is that any URL that comes
through in a DM gets currently wrapped with a twt.tl URL -- if the URL
turns out to be malicious, Twitter can simply shut it down, and whoever
follows that link will be presented with a page that warns them of
potentially malicious content. in a few weeks, we're going to start slowly
enabling this throughout the API for all statuses as well, but instead of
twt.tl, we will be using t.co.
practically, any tweet that is sent through statuses/update that has a
link on it will have the link automatically converted to a t.co link on
its way through the Twitter platform. if you fetch any tweet created after
this change goes live, then its text field will have all its links
automatically wrapped with t.co links. when a user clicks on that link,
Twitter will redirect them to the original URL after first confirming with
our database that that URL is not malicious. on top of the end-user
benefit, we hope to eventually provide all developers with aggregate usage
data around your applications such as the number of clicks people make on
URLs you display (it will, of course, be in aggregate and not identifiable
manner). additionally, we want to be able to build services and APIs that
can make algorithmic recommendations to users based on the content they are
consuming. gathering the data from t.co will help make these possible.
our current plan is that no user will see a t.co URL on twitter.com but
we still have some details to work through. the links will still be
displayed as they were sent in, but the target of the link will be the
t.co link instead. and, we want to provide the same ability to display
original links to developers. we're going to use the entities attribute to
make this possible.
let's say i send out the following tweet: you have to check out
http://dev.twitter.com!;
a returned (and truncated) status object may look like:
{
text : you have to check out http://t.co/s9gfk2d4!;,
...
user : {
screen_name : raffi,
...
},
...
entities : {
urls : [
{
url : http://t.co/s9gfk2d4;,
display_url : http://dev.twitter.com;,
indices : [23, 43]
}
],
...
},
...
}
two things to note: the text of the returned status object doesn't have
the original URL and instead it has a t.co URL, and the entities block
now has a display_url
[twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Raffi: Never mind. I just saw the Twitter blog post. The motivation for this is to get metrics for Promoted Tweets and Resonance. Hence, the answer is: Suck it up. DeWitt: Yikes, discarding all shortens between t.co and the final link will seriously mess with the click stats of a few million people.
Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API
Hi Raffi, On 9/06/10 9:57 AM, Raffi Krikorian wrote: that would be an awesome service! Currently we use one our own services (http://metauri.com/) to do this for http://trendsmap.com/. In addition to the title, it also gives the content type, which can be useful in determining how, or if to use the link in your display. I was just wondering if Twitter were going to possibly supply this extra data, as it would remove a time resource intensive step in the tweet analysis process :). Another question, will you wrap links that have no protocol in tweets, eg : Hey check out www.mysite.com/evil_page and : Hey check out mysite.com/evil_page I imagine many clients will currently link these out as urls and link them up automatically? Thanks again, JB. On Tue, Jun 8, 2010 at 4:50 PM, John Barratt djo...@gmail.com mailto:djo...@gmail.com wrote: Hi Raffi, On 9/06/10 8:57 AM, Raffi Krikorian wrote: url : http://t.co/s9gfk2d4;, display_url : http://dev.twitter.com;, indices : [23, 43] Any chance of getting the title of the resolved URL added in here too if available? Then we could display a link like : a title=Twitter Dev href=http://t.co/s9gfk2d4; http://dev.twitter.com /a or : a title=http://dev.twitter.com; href=http://t.co/s9gfk2d4; Twitter Dev /a This would give even more context to users, without having to follow the redirect path, load and parse the page to extract it as well. Thanks, JB. -- Raffi Krikorian Twitter Platform Team http://twitter.com/raffi
