Hi,

I've read Sean Burke's book, I've looked through the archives of this list and done other searches but can't find an answer to a problem I have found with LWP. If the character coding for a website has a byte order mark (things like utf-16, all that "big endian/little endian" stuff) then LWP can't interpret HTML headers in the usual way. Does anyone know a way around this?

Background:

I work for an organisation called ICRA. We provide a self-labelling and filtering system for the web, currently based on the old PICS standard but soon to move to RDF. A couple of years ago I built a tool for our website that visits a site, checks for PICS labels and parses them if found. Now, I can strip out the BOM from the content where found and do other clunky processing but that would mean I can't use LWP's efficient header commands. For sites without a BOM I can just get header->('Pics-label') and process that.

You can see the label tester at www.icra.org/label/tester/

An example of a site with a BOM that shows as unlabelled even though it is:
http://www.icra.org/cgi-bin/labelTester.cgi?lang=EN&url=http%3A%2F%2Fwww.xtranslations.com&showHead=on&showContent=on

An example of a site with a label without a BOM (i.e one that works as it should) would be http://www.icra.org/cgi-bin/labelTester.cgi?lang=EN&url=http%3A%2F%2Fwww.yahoo.com&showHead=on&showContent=on

Any help gratefully accepted.

Phil.

Phil Archer
Chief Technical Officer
Internet Content Rating Association
Label your site today at http://www.icra.org




Reply via email to