[twitter-dev] Re: Twitter Search issue
Hi Paul, It sounds like whatever is generating your API requests is double URL encoding. So the space becomes %20, and then on the second url encoding the % becomes a %25. Thanks; — Matt Sanford / @mzsanford On Mar 3, 2009, at 07:34 AM, Paul Kinlan wrote: Hi, I am noticing something that I think is odd at the moment. Some of our users are not getting searches that are enclosed in quotes via the API, yet they work directly from the website. For example there is a difference between the following query on the API and Website: http://search.twitter.com/search?q=%22exeter%20city%22 which has the same results as http://search.twitter.com/search?q=%22exeter+city%22 but returns a different result via the API using the following query http://search.twitter.com/search.json?q=exeter%20city; Looking at what is returned by the API the query looks like it has been transformed in to %22exeter%2520city%22. To me the %2520 looks odd when I would expect %20 Kind Regards, Paul Kinlan
[twitter-dev] Re: Twitter Search issue
Hi Matt,
I was typing the search term through IE (to test it after reports that
enclosed searches aren't working) as
http://search.twitter.com/search.json?q=exeter city which it then converts
to http://search.twitter.com/search.json?q=exeter%20city; but the result
came back as %22exeter*%2520*city%22 (see json object below) in the search
API json object. It works in firefox so I am presuming firefox is correctly
encoding the url.
{results:[],since_id:0,max_id:1273765306,refresh_url:?since_id=1273765306q=%22exeter%2520city%22,results_per_page:15,completed_in:1.313905,page:1,query:%22exeter%2520city%22}
it is highly likely that if IE is having the issue, the client API would
probably have it, however the query that is going out over the wire (I
checked with fiddler as exeter%20city and the result comes back as above,
so I don't think it is us for the entire problem).
Kind Regards,
Paul.
2009/3/3 Matt Sanford m...@twitter.com
Hi Paul,
It sounds like whatever is generating your API requests is double URL
encoding. So the space becomes %20, and then on the second url encoding the
% becomes a %25.
Thanks;
— Matt Sanford / @mzsanford
On Mar 3, 2009, at 07:34 AM, Paul Kinlan wrote:
Hi,
I am noticing something that I think is odd at the moment.
Some of our users are not getting searches that are enclosed in quotes via
the API, yet they work directly from the website.
For example there is a difference between the following query on the API
and Website:
http://search.twitter.com/search?q=%22exeter%20city%22 which has the same
results as http://search.twitter.com/search?q=%22exeter+city%22
but returns a different result via the API using the following query
http://search.twitter.com/search.json?q=exeter%20city;
Looking at what is returned by the API the query looks like it has been
transformed in to %22exeter*%2520*city%22. To me the %2520 looks odd
when I would expect %20
Kind Regards,
Paul Kinlan
[twitter-dev] Re: Twitter Search issue
I also just tested searching exeter city in TweetGrid with IE,
FireFox, and Chrome. All came back with the same results.
fwiw,
-Chad
On Tue, Mar 3, 2009 at 11:14 AM, Matt Sanford m...@twitter.com wrote:
Hi Paul,
I just tested form the command line and everything seems fine with: curl
'http://search.twitter.com/search.json?q=%22exeter%20city%22'
If you are typing %20 into the IE address bar it is likely try to
correct your % (which is not a valid URL character) and making it %25 in
the request but displaying it correctly to you. Try replacing it with a + or
a space and see what you get.
Thanks;
— Matt
- Show quoted text -
On Mar 3, 2009, at 08:06 AM, Paul Kinlan wrote:
Forgot to add, I am checking our client library now too.
Paul.
2009/3/3 Paul Kinlan paul.kin...@gmail.com
Hi Matt,
I was typing the search term through IE (to test it after reports that
enclosed searches aren't working) as
http://search.twitter.com/search.json?q=exeter city which it then converts
to http://search.twitter.com/search.json?q=exeter%20city; but the result
came back as %22exeter%2520city%22 (see json object below) in the search
API json object. It works in firefox so I am presuming firefox is correctly
encoding the url.
{results:[],since_id:0,max_id:1273765306,refresh_url:?since_id=1273765306q=%22exeter%2520city%22,results_per_page:15,completed_in:1.313905,page:1,query:%22exeter%2520city%22}
it is highly likely that if IE is having the issue, the client API would
probably have it, however the query that is going out over the wire (I
checked with fiddler as exeter%20city and the result comes back as above,
so I don't think it is us for the entire problem).
Kind Regards,
Paul.
2009/3/3 Matt Sanford m...@twitter.com
Hi Paul,
It sounds like whatever is generating your API requests is double URL
encoding. So the space becomes %20, and then on the second url encoding the
% becomes a %25.
Thanks;
— Matt Sanford / @mzsanford
On Mar 3, 2009, at 07:34 AM, Paul Kinlan wrote:
Hi,
I am noticing something that I think is odd at the moment.
Some of our users are not getting searches that are enclosed in quotes
via the API, yet they work directly from the website.
For example there is a difference between the following query on the API
and Website:
http://search.twitter.com/search?q=%22exeter%20city%22 which has the same
results as http://search.twitter.com/search?q=%22exeter+city%22
but returns a different result via the API using the following query
http://search.twitter.com/search.json?q=exeter%20city;
Looking at what is returned by the API the query looks like it has been
transformed in to %22exeter%2520city%22. To me the %2520 looks odd when I
would expect %20
Kind Regards,
Paul Kinlan
[twitter-dev] Re: Twitter Search issue
Hi,
It works with the +, but I knew that :)
With a space (in IE) it encodes it as %20 when it makes the request and I
can see it through fiddler (as below) and it comes back.
GET /search.json?q=exeter%20city HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-ms-application, application/vnd.ms-xpsdocument,
application/xaml+xml, application/x-ms-xbap, application/x-shockwave-flash,
application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword,
*/*
Accept-Language: en-gb
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618;
InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
Host: search.twitter.com
Connection: Keep-Alive
Cookie:
__utma=43838368.379476752167577530.1234449205.1234449205.1234449205.1;
__utmz=43838368.1234449205.1.1.utmcsr=blog.twe2.com|utmccn=(referral)|utmcmd=referral|utmcct=/;
__utmv=43838368.lang%3A%20en_GB
I fully accept it is probably our client software that is not encoding
correcly, but I also tried this from the command line curl
http://search.twitter.com/search.json?q=\exeter%20city\; the response
comes back as %22exeter%2520city%22 in the json object. From my point of
view I know the quotes are not correct, but it looks like twitter is
encoding them when it recieves them. I belive our client API is sending
double quotes rather %22.
Kind Regards,
Paul
2009/3/3 Chad Etzel jazzyc...@gmail.com
I also just tested searching exeter city in TweetGrid with IE,
FireFox, and Chrome. All came back with the same results.
fwiw,
-Chad
On Tue, Mar 3, 2009 at 11:14 AM, Matt Sanford m...@twitter.com wrote:
Hi Paul,
I just tested form the command line and everything seems fine
with: curl
'http://search.twitter.com/search.json?q=%22exeter%20city%22'
If you are typing %20 into the IE address bar it is likely try to
correct your % (which is not a valid URL character) and making it %25 in
the request but displaying it correctly to you. Try replacing it with a +
or
a space and see what you get.
Thanks;
— Matt
- Show quoted text -
On Mar 3, 2009, at 08:06 AM, Paul Kinlan wrote:
Forgot to add, I am checking our client library now too.
Paul.
2009/3/3 Paul Kinlan paul.kin...@gmail.com
Hi Matt,
I was typing the search term through IE (to test it after reports that
enclosed searches aren't working) as
http://search.twitter.com/search.json?q=exeter city which it then
converts
to http://search.twitter.com/search.json?q=exeter%20city; but the
result
came back as %22exeter%2520city%22 (see json object below) in the
search
API json object. It works in firefox so I am presuming firefox is
correctly
encoding the url.
{results:[],since_id:0,max_id:1273765306,refresh_url:?since_id=1273765306q=%22exeter%2520city%22,results_per_page:15,completed_in:1.313905,page:1,query:%22exeter%2520city%22}
it is highly likely that if IE is having the issue, the client API would
probably have it, however the query that is going out over the wire (I
checked with fiddler as exeter%20city and the result comes back as
above,
so I don't think it is us for the entire problem).
Kind Regards,
Paul.
2009/3/3 Matt Sanford m...@twitter.com
Hi Paul,
It sounds like whatever is generating your API requests is double
URL
encoding. So the space becomes %20, and then on the second url encoding
the
% becomes a %25.
Thanks;
— Matt Sanford / @mzsanford
On Mar 3, 2009, at 07:34 AM, Paul Kinlan wrote:
Hi,
I am noticing something that I think is odd at the moment.
Some of our users are not getting searches that are enclosed in quotes
via the API, yet they work directly from the website.
For example there is a difference between the following query on the
API
and Website:
http://search.twitter.com/search?q=%22exeter%20city%22 which has the
same
results as http://search.twitter.com/search?q=%22exeter+city%22
but returns a different result via the API using the following query
http://search.twitter.com/search.json?q=exeter%20city;
Looking at what is returned by the API the query looks like it has been
transformed in to %22exeter%2520city%22. To me the %2520 looks odd
when I
would expect %20
Kind Regards,
Paul Kinlan
[twitter-dev] Re: Twitter Search issue
Ok, I can replicate your results with curl
$ curl -v http://search.twitter.com/search.json?q=\exeter%20city\;
...returns the wrong results, as you say.
$ curl -v http://search.twitter.com/search.json?q=%22exeter%20city%22;
...returns the correct results.
I think double quotes are not actually valid URL characters (tho some
browsers try to treat them as such), so you should really turn into
%22 before the requests go out.
That said, I'm starting to agree with Paul that twitter is doing some
sort of encoding trick on their end when literal quotes are sent in
the request:
$ curl -v http://search.twitter.com/search.json?q=\exeter%20city\;
* About to connect() to search.twitter.com port 80 (#0)
* Trying 128.121.146.107... connected
* Connected to search.twitter.com (128.121.146.107) port 80 (#0)
GET /search.json?q=exeter%20city HTTP/1.1
User-Agent: curl/7.16.4 (i486-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3.3 libidn/1.0
Host: search.twitter.com
Accept: */*
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2009 17:09:34 GMT
Server: hi
Status: 200 OK
Cache-Control: max-age=20, must-revalidate, max-age=300
Content-Type: application/json; charset=utf-8
X-Served-By: searchweb003.twitter.com
Expires: Tue, 03 Mar 2009 17:14:34 GMT
Content-Length: 195
Vary: Accept-Encoding
X-Varnish: 1733084231
Age: 0
Via: 1.1 varnish
X-Cache-Svr: searchweb003.twitter.com
X-Cache: MISS
Connection: close
* Closing connection #0
{results:[],since_id:0,max_id:1274236746,refresh_url:?since_id=1274236746q=%22exeter%2520city%22,results_per_page:15,completed_in:0.112164,page:1,query:%22exeter%2520city%22}
Now, one could argue that the request itself is invalid or malformed,
and so the result may be undefined, but I do agree that something is
happening on twitter's end.
Moral of the story: encode as %22 in URLs.
-Chad
On Tue, Mar 3, 2009 at 11:33 AM, Paul Kinlan paul.kin...@gmail.com wrote:
Hi,
It works with the +, but I knew that :)
With a space (in IE) it encodes it as %20 when it makes the request and I
can see it through fiddler (as below) and it comes back.
GET /search.json?q=exeter%20city HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-ms-application, application/vnd.ms-xpsdocument,
application/xaml+xml, application/x-ms-xbap, application/x-shockwave-flash,
application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword,
*/*
Accept-Language: en-gb
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618;
InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
Host: search.twitter.com
Connection: Keep-Alive
Cookie:
__utma=43838368.379476752167577530.1234449205.1234449205.1234449205.1;
__utmz=43838368.1234449205.1.1.utmcsr=blog.twe2.com|utmccn=(referral)|utmcmd=referral|utmcct=/;
__utmv=43838368.lang%3A%20en_GB
I fully accept it is probably our client software that is not encoding
correcly, but I also tried this from the command line curl
http://search.twitter.com/search.json?q=\exeter%20city\; the response
comes back as %22exeter%2520city%22 in the json object. From my point of
view I know the quotes are not correct, but it looks like twitter is
encoding them when it recieves them. I belive our client API is sending
double quotes rather %22.
Kind Regards,
Paul
2009/3/3 Chad Etzel jazzyc...@gmail.com
I also just tested searching exeter city in TweetGrid with IE,
FireFox, and Chrome. All came back with the same results.
fwiw,
-Chad
On Tue, Mar 3, 2009 at 11:14 AM, Matt Sanford m...@twitter.com wrote:
Hi Paul,
I just tested form the command line and everything seems fine
with: curl
'http://search.twitter.com/search.json?q=%22exeter%20city%22'
If you are typing %20 into the IE address bar it is likely try to
correct your % (which is not a valid URL character) and making it %25
in
the request but displaying it correctly to you. Try replacing it with a
+ or
a space and see what you get.
Thanks;
— Matt
- Show quoted text -
On Mar 3, 2009, at 08:06 AM, Paul Kinlan wrote:
Forgot to add, I am checking our client library now too.
Paul.
2009/3/3 Paul Kinlan paul.kin...@gmail.com
Hi Matt,
I was typing the search term through IE (to test it after reports that
enclosed searches aren't working) as
http://search.twitter.com/search.json?q=exeter city which it then
converts
to http://search.twitter.com/search.json?q=exeter%20city; but the
result
came back as %22exeter%2520city%22 (see json object below) in the
search
API json object. It works in firefox so I am presuming firefox is
correctly
encoding the url.
{results:[],since_id:0,max_id:1273765306,refresh_url:?since_id=1273765306q=%22exeter%2520city%22,results_per_page:15,completed_in:1.313905,page:1,query:%22exeter%2520city%22}
it is highly
[twitter-dev] Re: Twitter Search issue
Hi,
Yeah, I am pretty sure our Api client takes a litteral query string and
since we store it that way it is probablly sending it that way.
Paul.
2009/3/3 Matt Sanford m...@twitter.com
Hi all,
If you send something invalid we do attempt to fix-up invalid requests
rather than just 400. This looks like a case where the bad request becomes a
different sort of badness on the way out. Escaping the quotes seems like the
only real fix.
— Matt
On Mar 3, 2009, at 09:13 AM, Chad Etzel wrote:
Ok, I can replicate your results with curl
$ curl -v http://search.twitter.com/search.json?q=\exeter%20city\;
...returns the wrong results, as you say.
$ curl -v http://search.twitter.com/search.json?q=%22exeter%20city%22;
...returns the correct results.
I think double quotes are not actually valid URL characters (tho some
browsers try to treat them as such), so you should really turn into
%22 before the requests go out.
That said, I'm starting to agree with Paul that twitter is doing some
sort of encoding trick on their end when literal quotes are sent in
the request:
$ curl -v http://search.twitter.com/search.json?q=\exeter%20city\;
* About to connect() to search.twitter.com port 80 (#0)
* Trying 128.121.146.107... connected
* Connected to search.twitter.com (128.121.146.107) port 80 (#0)
GET /search.json?q=exeter%20city HTTP/1.1
User-Agent: curl/7.16.4 (i486-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3.3 libidn/1.0
Host: search.twitter.com
Accept: */*
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2009 17:09:34 GMT
Server: hi
Status: 200 OK
Cache-Control: max-age=20, must-revalidate, max-age=300
Content-Type: application/json; charset=utf-8
X-Served-By: searchweb003.twitter.com
Expires: Tue, 03 Mar 2009 17:14:34 GMT
Content-Length: 195
Vary: Accept-Encoding
X-Varnish: 1733084231
Age: 0
Via: 1.1 varnish
X-Cache-Svr: searchweb003.twitter.com
X-Cache: MISS
Connection: close
* Closing connection #0
{results:[],since_id:0,max_id:1274236746,refresh_url:?since_id=1274236746q=%22exeter%2520city%22,results_per_page:15,completed_in:0.112164,page:1,query:%22exeter%2520city%22}
Now, one could argue that the request itself is invalid or malformed,
and so the result may be undefined, but I do agree that something is
happening on twitter's end.
Moral of the story: encode as %22 in URLs.
-Chad
On Tue, Mar 3, 2009 at 11:33 AM, Paul Kinlan paul.kin...@gmail.com
wrote:
Hi,
It works with the +, but I knew that :)
With a space (in IE) it encodes it as %20 when it makes the request and I
can see it through fiddler (as below) and it comes back.
GET /search.json?q=exeter%20city HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-ms-application, application/vnd.ms-xpsdocument,
application/xaml+xml, application/x-ms-xbap, application/x-shockwave-flash,
application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword,
*/*
Accept-Language: en-gb
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618;
InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
Host: search.twitter.com
Connection: Keep-Alive
Cookie:
__utma=43838368.379476752167577530.1234449205.1234449205.1234449205.1;
__utmz=43838368.1234449205.1.1.utmcsr=blog.twe2.com
|utmccn=(referral)|utmcmd=referral|utmcct=/;
__utmv=43838368.lang%3A%20en_GB
I fully accept it is probably our client software that is not encoding
correcly, but I also tried this from the command line curl
http://search.twitter.com/search.json?q=\exeter%20city\; the response
comes back as %22exeter%2520city%22 in the json object. From my point of
view I know the quotes are not correct, but it looks like twitter is
encoding them when it recieves them. I belive our client API is sending
double quotes rather %22.
Kind Regards,
Paul
2009/3/3 Chad Etzel jazzyc...@gmail.com
I also just tested searching exeter city in TweetGrid with IE,
FireFox, and Chrome. All came back with the same results.
fwiw,
-Chad
On Tue, Mar 3, 2009 at 11:14 AM, Matt Sanford m...@twitter.com wrote:
Hi Paul,
I just tested form the command line and everything seems fine
with: curl
'http://search.twitter.com/search.json?q=%22exeter%20city%22'http://search.twitter.com/search.json?q=%22exeter%20city%22%27
If you are typing %20 into the IE address bar it is likely try to
correct your % (which is not a valid URL character) and making it %25
in
the request but displaying it correctly to you. Try replacing it with a
+ or
a space and see what you get.
Thanks;
— Matt
- Show quoted text -
On Mar 3, 2009, at 08:06 AM, Paul Kinlan wrote:
Forgot to add, I am checking our client library now too.
Paul.
2009/3/3 Paul Kinlan paul.kin...@gmail.com
Hi Matt,
I
