Hi,
I've read Sean Burke's book, I've looked through the archives of this list and done other searches but can't find an answer to a problem I have found with LWP. If the character coding for a website has a byte order mark (things like utf-16, all that "big endian/little endian" stuff) then LWP can't interpret HTML headers in the usual way. Does anyone know a way around this?
Background:
I work for an organisation called ICRA. We provide a self-labelling and filtering system for the web, currently based on the old PICS standard but soon to move to RDF. A couple of years ago I built a tool for our website that visits a site, checks for PICS labels and parses them if found. Now, I can strip out the BOM from the content where found and do other clunky processing but that would mean I can't use LWP's efficient header commands. For sites without a BOM I can just get header->('Pics-label') and process that.
You can see the label tester at www.icra.org/label/tester/
An example of a site with a BOM that shows as unlabelled even though it is: http://www.icra.org/cgi-bin/labelTester.cgi?lang=EN&url=http%3A%2F%2Fwww.xtranslations.com&showHead=on&showContent=on
An example of a site with a label without a BOM (i.e one that works as it should) would be http://www.icra.org/cgi-bin/labelTester.cgi?lang=EN&url=http%3A%2F%2Fwww.yahoo.com&showHead=on&showContent=on
Any help gratefully accepted.
Phil.
Phil Archer Chief Technical Officer Internet Content Rating Association Label your site today at http://www.icra.org
