On 10-11-20 02:54 PM, José Romildo Malaquias wrote:
In order to download a given web page, I wrote the attached program. The
problem is that the page is not being full downloaded. It is being
somehow intettupted.

The specific website and url
http://www.adorocinema.com/common/search/search_by_film/?criteria=Bourne

truncates when the web server chooses the identity encoding (i.e., as opposed to compressed ones such as gzip). The server chooses identity when your request's Accept-Encoding field specifies identity or simply your request has no Accept-Encoding field, such as when you use simpleHTTP (getRequest url), curl, wget, elinks.

When the server chooses gzip (its favourite), which is when your Accept-Encoding field includes gzip, the received data is complete (but then you have to gunzip it yourself). This happens with mainstream browsers and W3C's validator at validator.w3.org (which destroys the "you need javascript" hypothesis). I haven't tested other compressed encodings.

Methodology

My methodology of discovering and confirming this is a great lesson in the triumph of the scientific methodology (over the prevailing opinionative methodology, for example).

The first step is to confirm or deny a Network.HTTP problem. For a maximally controlled experiment, I enter HTTP by hand using nc:

$ nc www.adorocinema.com 80
GET /common/search/search_by_film/?criteria=Bourne HTTP/1.1
Host: www.adorocinema.com
<blank line>

It still truncates, so at least Network.HTTP is not alone. I also try elinks. Other people try curl and wget for the same reason and the same result.

The second step is to confirm or deny javascript magic. Actually the truncation strongly suggests that javascript is not involved: the truncation ends with an incomplete end-tag "</". This is abnormal even for very buggy javascript-heavy web pages. To certainly deny javascript magic, I first try Firefox with javascript off (also java off, flash off, even css off), and then I also ask validator.w3.org to validate the page. Both receive complete data. Of course the validator is going to say "many errors", but the point is that if the validator reports errors at locations way beyond our truncation point, then the validator sees data we don't see, and the validator doesn't even care about javascript.

The validator may be very sophisticated in parsing html, but in sending an HTTP request it ought to be very simple-minded. The third step is to find out what extra thing the validator does to deserve complete data. So I try diagonalization: I give this CGI script to the validator:

#! /bin/sh

echo 'Content-Type: text/html'
echo ''
e=`env`
cat <<EOF
<html><head><title>title</title></head><body><pre>
$e
</pre></body></html>
EOF

If I also tell the validator to "show source", which means display what html code it sees, then I see the validator's request (barring web server support, but modern web servers probably support much more than the original minimal CGI specification). There are indeed quite a few extra header fields the validator sends, and I can try to mimic each of them. Eventually I find this crucial field:

Accept-Encoding: gzip, x-gzip, deflate

Further tests confirm that we just need gzip.

Finally, to confirm the finding with a maximally controlled experiment, I enter the improved request by hand using nc, but this time I save the output in a file (so later I can decompress it):

$ nc -q 10 www.adorocinema.com 80 > save.gz
GET /common/search/search_by_film/?criteria=Bourne HTTP/1.1
Host: www.adorocinema.com
Accept-Encoding: gzip
<blank line>
<wait a while>

Now save.gz contains both header and body, and it only makes sense to uncompress the body. So edit save.gz to delete the header part.

Applying gunzip to the body will give some "unexpected end of file" error. Don't despair. Do this instead:

$ zcat save.gz > save.html

It is still an error but save.html has meaningful and complete content. You can examine it. You can load it in a web browser and see. At least, it is much longer, and it ends with "</html>" rather than "</".

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Reply via email to