It'd be rather easy to wrap the streams in a DeflaterOutputStream or an
InflaterInputStream. Of course, due to limitations in Java's deflate
compression, one must extend DeflaterOutputStream to allow true stream
deflation. The problem with the current implementation is that there is
no way to flush a partially deflated stream -- deflate waits until it
reaches an optimal spot to actually perform the deflation and do the
flush. You can see
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html and
http://developer.java.sun.com/developer/bugParade/bugs/4206909.html .
In LimeWire we worked around this by using a workaround listed on those
pages: having a class 'CompressingOutputStream' that extends
DeflaterOutputStream, using the following as the flush method:
private static final byte [] EMPTYBYTEARRAY = new byte [0];
/**
* Ensure all remaining data will be output.
*/
public void flush() throws IOException {
if( def.finished() ) return;
/**
* Now this is tricky: We force the Deflater to flush
* its data by switching compression level.
* As yet, a perplexingly simple workaround for
*
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html
*/
def.setInput(EMPTYBYTEARRAY, 0, 0);
def.setLevel(Deflater.NO_COMPRESSION);
deflate();
def.setLevel(Deflater.DEFAULT_COMPRESSION);
deflate();
super.flush();
}
The other thing you have to be careful of when using
DeflaterOutputStream or InflaterInputStream is spurious
NullPointerExceptions from native code if the connection happens to
close while a deflate/inflate is being performed. We worked around that
particular problem by extending the 'read' of the InflaterInputStream
and the 'deflate' call of the DeflaterOutputStream to catch an NPE and
rethrow an IOX.
Thanks,
Sam
otisg wrote:
I'm not too familiar with HttpClient source code any more, but
it seems to me that this should be easy to add. Please correct
me if I am wrong.
I will be using HttpClient to pull large amounts of data from
the web in a few months, and allowing compressed content may
translate to less bandwidth and a lower hosting bill :)
Otis
Get your own 800 number
Voicemail, fax, email, and a lot more
http://www.ureach.com/reg/tag
On Mon, 17 Nov 2003, =?ISO-8859-15?Q?Sven_K=F6hler?=
([EMAIL PROTECTED]) wrote:
Hi,
If i send the right accept-encoding headers, the web-server
may answer
with a gzip or deflate compressed stream. Does HttpClient
decompress it?
If yes, how can i turn that off?
-
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
