I have no smoking gun to indicate this is a mechanize problem. just instinct. I use mechanize daily as part of my job and its an old dear friend =)
current problem: for a particular site, for a particular request, the response i get with mechanize is extremely truncated in the body of the response. the headers seems to come back fine, but instead of getting 20k of response i get 2700 bytes or so. comparing it to a view source from a browser, they are line for line identical - until the mechanize object simply ends as if it gets an EOF too early, or some control character or something. does anyone have advice on debugging this? I am using a modified version of Mechanize thats a bit old, so if there have been any known problems in the past with truncation, well im probably not using their fixes. any advice on how to debug this? heres a sample of what I mean. We start to get some correct html, but then it just stops about 10 percent in. you will see it get a few lines of javascript then suddenly end the response. from putting the same url (non edited) into a web browser, the page is ten times larger. the line right after the </script> tag (the last we get) is a blank line. # COMMENT: From [some site] via submit() $response = bless( { '_protocol' => 'HTTP/1.1', '_content' => '<!-- Generated by WebSpeed - http://www.webspeed.com/ --> <html> <HTML> <SCRIPT> var mykey; </SCRIPT> <SCRIPT LANGUAGE="JavaScript"> function updateVendor(newVend) { var i; for(i = 0; i < parent.frames[0].document.user.selectedvendor.options.length; i++) { if (parent.frames[0].document.user.selectedvendor.options[i].value == newVend) { parent.frames[0].document.user.selectedvendor.options.selectedIndex = i; parent.frames[0].document.user.submit(); break; } } } </SCRIPT> <script language="JavaScript" type="text/JavaScript"> <!-- var monthsF = new Array (31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31); function doLeapYearF (inY) { if ((inY % 400 == 0) || (inY % 4 == 0) && (!(inY % 100 == 0))) { monthsF[1] = 29; } else { monthsF[1] = 28; } } function setFDays (ipt, mn, inY) { var sl = 0; if (ipt.selectedIndex) { sl = ipt.selectedIndex; } doLeapYearF (inY); ipt.length = 0; for (var i = 0; i < monthsF[mn-1]; i++) { ipt[ipt.length] = new Option (i+1, i+1); } if (sl > (ipt.length-1)) { sl = (ipt.length-1); } ipt.options[sl].selected = true; } function modFDays () { with (document.details) { setFDays (slFromDay, slFromMonth[slFromMonth.selectedIndex].value, slFromYear[slFromYear.selectedIndex].valu e); setFDays (slToDay, slToMonth[slToMonth.selectedIndex].value, slToYear[slToYear.selectedIndex].value); } } //--> </script> ', '_rc' => '200', '_headers' => bless( { 'connection' => 'close', 'client-response-num' => 1, 'date' => 'Mon, 22 Dec 2003 15:52:41 GMT', 'vary' => 'Accept-Encoding', 'client-peer' => '62.254.203.12:80', 'client-date' => 'Mon, 22 Dec 2003 15:51:52 GMT', 'content-type' => 'text/html', 'server' => 'Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b mod_perl/1.26', 'client-transfer-encoding' => 'chunked' }, 'HTTP::Headers' ), '_msg' => 'OK', '_request' => bless( { '_content' => 'area=7', '_uri' => bless( do{\(my $o = 'http://www.somesite.com/cgi-bin/liveweb.sh/MainFrame.w?UckUcZllaaIpkkac=&POlfi gXkjkjkWbdq=&SblzdbFStskKccak=&RkkFlPbjklpsIaUb=FRA&aff=&ctryref=FRA%E2%8C%A9=FR&currcode=&ClerkID=')}, 'URI::http' ), '_headers' => bless( { 'user-agent' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)', 'content-type' => 'application/x-www-form-urlencoded', 'content-length' => 6, 'referer' => 'http://www.somesite.com/cgi-bin/liveweb.sh/MainFrame.w?ctryref=FRA&lang=FR' }, 'HTTP::Headers' ), '_method' => 'POST' }, 'HTTP::Request' ) }, 'HTTP::Response' ); -- gedanken