Oleg, This is exactly what I tried to do, but the HttpClient seems to buffer the response anyway. I poked through the code a bit, and did some searching on the mailing list, and this seems to be the case. Also, when I run tests, the log output from the HttpClient seems to indicate that the request is being buffered before I even get an opportunity to call "getResponseBodyAsStream". I tried this on a request that is about 17 MB and just sat back and watched as the response was buffered; I ended up terminating the program after a minute or two without ever actually getting past the execute method call. Here is the output from the logs:
enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection) enter GetMethod.readResponseBody(HttpState, HttpConnection) enter HttpMethodBase.readResponseBody(HttpState, HttpConnection) enter HttpMethodBase.readResponseBody(HttpState, HttpConnection) enter HttpConnection.getResponseInputStream() Buffering response body <lots of logs detailing the bytes read> >From here it looks like the "processResponseHeaders" method (which is called by the HttpMethodBase during an execute) automatically buffers the response. Is this true, or am I totally misreading the signs? Thanks, Bob -----Original Message----- From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 29, 2004 3:04 PM To: Commons HttpClient Project Subject: Re: streaming responses Bob, There's no special magic involved. Make sure you use HttpMethod#getResponseBodyAsStream, which will return the raw input stream, and not its buffering counterparts HttpMethod#getResponseBodyAsString and HttpMethod#getResponseBody. You should not worry about chunking. HttpClient will decode chunked input on fly, if necessary. Just grab the raw input stream and do whatever response retrieval suits your application best. Hope this helps Oleg On Wed, 2004-09-29 at 20:43, St Jacques, Robert wrote: > Howdy, > > I've just started using HttpClient for testing. The product that I am > developing includes a software download feature that downloads > (sometimes very large) files over the internet using HTTP. The reason > we use HTTP is that many of our customers are unwilling to open their > proxies or firewalls to other protocols. > > In order to accommodate these downloads, we must either be able to > stream the response to a GET, or we need to be able to insure that the > response we get from the server will be chunked in such a way that the > chunks are small enough to buffer. I know that HttpClient supports > chunking, but I am unable to 'force' our web server to chunk up the > data. Furthermore, the HTTP 1.1 RFC indicates that the server does > not need to guarantee that a response, even if chunked, will be small > enough to conveniently buffer on the client. > > What is the best way to use HttpClient to retrieve large amounts of > data (as large as 200 MB or more)? > > Thanks, > Bob St. Jacques > Member Research Technical Staff > Xerox Corp. > (585)231-8306 > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
