Re: Does HttpClient decompress compressed HTTP transfers?

[Sam Berlin]

It'd be rather easy to wrap the streams in a DeflaterOutputStream or an 
InflaterInputStream.  Of course, due to limitations in Java's deflate 
compression, one must extend DeflaterOutputStream to allow true stream 
deflation.  The problem with the current implementation is that there is 
no way to flush a partially deflated stream -- deflate waits until it 
reaches an optimal spot to actually perform the deflation and do the 
flush.  You can see  
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html and 
http://developer.java.sun.com/developer/bugParade/bugs/4206909.html .  
In LimeWire we worked around this by using a workaround listed on those 
pages: having a class 'CompressingOutputStream' that extends 
DeflaterOutputStream, using the following as the flush method:

   private static final byte [] EMPTYBYTEARRAY = new byte [0];

   /**
    * Ensure all remaining data will be output.
    */
   public void flush() throws IOException {
       if( def.finished() ) return;
      
       /**
        * Now this is tricky: We force the Deflater to flush
        * its data by switching compression level.
        * As yet, a perplexingly simple workaround for
        * 
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html
        */
       def.setInput(EMPTYBYTEARRAY, 0, 0);

       def.setLevel(Deflater.NO_COMPRESSION);
       deflate();
       def.setLevel(Deflater.DEFAULT_COMPRESSION);
       deflate();
       super.flush();
   }
The other thing you have to be careful of when using 
DeflaterOutputStream or InflaterInputStream is spurious 
NullPointerExceptions from native code if the connection happens to 
close while a deflate/inflate is being performed.  We worked around that 
particular problem by extending the 'read' of the InflaterInputStream 
and the 'deflate' call of the DeflaterOutputStream to catch an NPE and 
rethrow an IOX.

Thanks,
Sam
otisg wrote:

I'm not too familiar with HttpClient source code any more, but
it seems to me that this should be easy to add.  Please correct
me if I am wrong.
I will be using HttpClient to pull large amounts of data from
the web in a few months, and allowing compressed content may
translate to less bandwidth and a lower hosting bill :)
Otis



________________________________________________
Get your own "800" number
Voicemail, fax, email, and a lot more
http://www.ureach.com/reg/tag
---- On Mon, 17 Nov 2003, =?ISO-8859-15?Q?Sven_K=F6hler?=
([EMAIL PROTECTED]) wrote:
 

Hi,

If i send the right accept-encoding headers, the web-server
   

may answer 
 

with a gzip or deflate compressed stream. Does HttpClient
   

decompress it? 
 

If yes, how can i turn that off?



   

---------------------------------------------------------------------
 

To unsubscribe, e-mail:
   

[EMAIL PROTECTED]
 

For additional commands, e-mail:
   

[EMAIL PROTECTED]
 

   



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
.

 



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]