Unable to retreive all data of a large (>30 MB) file.

[Spencer Lee]

Hello,

Just wondering if anyone has had problems retreiving data all the data of a
large file.

Here's basically the code I've been running...

...
           HttpClient httpclient = new HttpClient(params);
           httpclient.getHostConfiguration().setHost("myhost", 80, "http");
           httpclient.getParams().setCookiePolicy(
CookiePolicy.BROWSER_COMPATIBILITY);
           httpclient.getHostConfiguration ().setProxy("webproxy", 80);
...
           // Create a method instance.
           GetMethod method = new GetMethod(myurl);
           DefaultHttpMethodRetryHandler retryhandler = new
DefaultHttpMethodRetryHandler(10, false);
           HttpMethodParams method_params = method.getParams();
           method_params.setParameter(HttpMethodParams.RETRY_HANDLER,
retryhandler);
...
           // executing the method
           int statusCode = httpclient.executeMethod(method);

           if(statusCode != HttpStatus.SC_OK) {
               System.out.println("ERROR: Status code " + statusCode);
               return;
           }

           // Read the response body.
           InputStream inpStream = method.getResponseBodyAsStream();

           BufferedInputStream bufinstrm = new
BufferedInputStream(inpStream);
           long totalBytesRead = 0;
           while(true) {
               byte[] buffer = new byte[12*1024];
               int bytesRead;

               bytesRead = bufinstrm.read(buffer);
               if(bytesRead == -1) {
                   System.err.println("Read -1.  Read complete.  Total
Bytes Read: " + totalBytesRead + " bytes");
                   break;
               }

   //            bufotstrm.write(buffer, 0, bytesRead);
               totalBytesRead += bytesRead;
               System.err.println("Total Bytes Read: " + totalBytesRead + "
bytes");
           }

           bufinstrm.close();

           // close
           method.releaseConnection();



The byte size of the file is 36,197,082.  But I am only able to retreival
all 36,197,082 bytes once in awhile.  On most occasions my
BufferedInputStream (bufinstrm) will return a -1 to signal EOF before all
bytes are read.

eg.
 ...
 Total Bytes Read: 26796032 bytes
 Read -1.  Read complete.  Total Bytes Read: 26796032 bytes
 >

Anyone come across anything similar to this before?

Thanks!

Spencer

Also...
* I'm using release commons-httpclient-3.0.1
* I am not seeing any abnormal messages printed to the logs.
* I am working through a web proxy server.