Hi all..
I have added a new directory 'org.mmbase.servlet.filter' with a few files for quickly
defining
new filters. I have also added 'GZipFilter.java' which can be chained between filters
and
will gzip the incoming request and send it clientside if it is defined as the last
filter.
This filter will reduce network traffic significantly (we see a drop of ~ 40% - 60%)
by Gzipping
the stream. Code contains documentation to install it, but for the impatient, add the
following
line to web.xml after compile:
...
<filter>
<filter-name>gzip</filter-name>
<display-name>GZIPFilter</display-name>
<filter-class>org.mmbase.servlet.filter.GZIPFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>gzip</filter-name>
<url-pattern>*</url-pattern>
</filter-mapping>
...
and add the following lines in log4j.xml:
...
<category name="org.mmbase.servlet.filter" class ='&logger;'>
<priority class='&priority;' value ="info" />
</category>
...
This should wotk for all jsp-engines like tomcat and orion.
Now for some real examples; if properly installed it should output lines in the
logfile like these:
20:51 INFO org.mmbase.servlet.filter.GZIPFilter: url(/cinema/index_bioscoop.jsp):
page(639161), gzip(31015), ratio(4 %)
.. which indicates that a 639K file has been obliterated to a mere 31k, which is 4% of
its total length before
sending it to the browser!! Pretty impressive with webdesigners like ours :)
The code excludes some requests, mostly 'gif' and 'jpg' files (images are included with
/img.db!!!) and some other odities like jump.db.
However, we noted some rather strange behaviour with realplayer (what else is new) in
that it
has some enormous trouble displaying gzip'ed smil-files. These are filtered out when
detected
(smil and ssmill files, rastreams and rmstreams, look in the code for examples).
Special care has also been taken care of when detecting <mm:includes> with a rather
dirty hack (checking if the agent is 'Java') so that includes on a page do not get
zipped
also (like zipping a zipfile). If somebody has a clever way as to detect this feature,
please
email to the list, since I currently do not see an easy way of detecting those, other
than
this check.
Also note that if the client indicates that it does not honour gzip'ed streams, it
will be
returned the 'normal' one, so noone should know that the data is compressed if the
browser does not understand it. MSIE on the mac is notorious for it, I've included a
list of Agents in the logfile as well as below.
Have fun saving those precious bits!
Marcel Maatkamp
VPRO Digitaal
---
Not honouring gzip agents:
DA 5.0
Eidetica/1.0 (http://eidetica.com/spider)
FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
LeechGet 2002 (www.leechget.de)
Lite Bot 0916b
MFC_Tear_Sample
MSProxy/2.0
Microsoft URL Control - 6.00.8169
Mozilla/3.0 (INGRID/3.0 MT; [EMAIL PROTECTED];
http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (compatible; AvantGo 3.2)
Mozilla/3.01 (compatible;)
Mozilla/3.01C-WXS-16 (Win95; I)
Mozilla/4.0 ( )
Mozilla/4.0 (compatible; MSIE 4.5; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; DT)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT)
Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; E503628FREELER1)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Trouw Internet)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; CNETHomeBuild051099; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; Tommy Hilfiger Europe b.v.)
Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.13; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.14; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.16; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.21; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.22; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 95)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; CHWIE_NL60)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; CWIE55NL60; CHWIE_NL60)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; HCCnet Internet Explorer)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Het Net 4.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Hotbar 4.1.8.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SKY11)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Supplied by blueyonder)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; T312461)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; CWIE55NL60; CHWIE_NL60;
H010818)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Hotbar 3.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; H010818)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; Unilever Research)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; TNET5.0NL)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; User attatched to NJIT's network in
the NJSOA domain)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; CHWIE_NL60)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; CHWIE_NL70)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Compaq; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Freegates:youri)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; HCCnet Internet Explorer)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; HCCnet)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Hotbar 4.1.8.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; MSN 6.1; MSNbMSFT; MSNmnl-be; MSNc00)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; MSN 6.1; MSNbMSFT; MSNmnl-be; MSNc0z)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; QXW0339t)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; TNET5.0NL)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; TUCOWS)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; WELZORG)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; YComp 5.0.0.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; c't)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; {Tiscali}; MSN 6.1; MSNbMSFT;
MSNmnl-nl; MSNc00)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; {Tiscali}; MSN 6.1; MSNbMSFT;
MSNmnl-nl; MSNc0z)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) WebWasher 3.2
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; (R1 1.1))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; CHWIE_BE70)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; DigExt; CHWIE_NL70)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; vnunet 5.5 200012)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)(R1 1.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AIRF)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AtHome033)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CHWIE_NL60)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CHWIE_NL70)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CWIE55NL60; CHWIE_NL60; MSN 6.1;
MSNbMSFT; MSNmnl-nl; MSNcOTH)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Het Net 3.2)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSN 6.1; MSNbMSFT; MSNmnl-be;
MSNcOTH)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSOCD)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSOCD; AIRF)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; TNET5.0NL)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 5.0.0.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 5.0.0.0; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Zon Breedband)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; {Tiscali})
Mozilla/5.0 (Slurp/cat; [EMAIL PROTECTED]; http://www.inktomi.com/slurp.html)
Mozilla/5.5 (compatible; MSIE 5.5)
NSPlayer/4.1.0.3856
Netscape 3.0 Compatible (WhatsNew Robot)
RMA/1.0 (compatible; RealMedia)
RealPlayer 4.0
RealPlayer 5.0
Scooter/3.2.SF0
Windows-Media-Player/8.00.00.4477
Zoek.nl N/G 1.0
contype
http://www.almaden.ibm.com/cs/crawler [c01]
ia_archiver
larbin_2.6.2 [EMAIL PROTECTED]
none
sharp-tq-gx10/1.0 profile/midp-1.0 configuration/cldc-1.0 up.browser/6.1.0.3.121c
(gui) mmp/1.0
sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)
