Hi,
I've been working on a hobby project of mine and recently I decided to
opensource it.
https://github.com/nielsbasjes/logparser
What this does is that it takes the LogFormat specification directly from
the httpd config that was used to write the file, and construct a parser
that does the inverse (parsing) operation. So instead of reinventing the
most horrendus regular expression ever when someone decides to change the
logformat you can now simply copy the new LogFormat specification and
continue with minimal downtime.
The codebase includes
- an Pojo/Annotation based parser that can be used in regular Java
applications,
- a Hadoop InputFormat for use in Java MapReduce applciations
- a Pig Loader.
As said, this is a hobby project of mine and I'm using (mostly the Pig
Loader) for several of my own experiments.
Because I think it's "the best thing since sliced bread" I've become kind
of blind for the flaws.
So I would really appreciate it if you guys can let me know what you think.
Thanks.
Niels Basjes
====================================================
Quote from the readme:
*Usage (PIG)
*
You simply register the httpdlog-pigloader-1.0-SNAPSHOT-job.jar
*REGISTER target/httpdlog-pigloader-1.0-SNAPSHOT-job.jar
*
And then call the loader with a dummy file (must exist, won't be read) and
the parameter called 'fields'. This will return a list of all possible
fields. Note that wher a '*' appears this means there are many possible
values that can appear there (for example the keys of a query string in a
URL). As you can see there is a kinda sloppy type mechanism to stear the
parsing, don't change that as the persing really relies on this.
*Fields =
LOAD 'test.pig' -- Any file as long as it exists
USING nl.basjes.pig.input.apachehttpdlog.Loader(
'%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"',
'Fields' ) AS (fields);
DESCRIBE Fields;
DUMP Fields;
*
Now that we have all the possible values that CAN be produced from this
logformat we simply choose the ones we need and tell the Loader we want
those.
*Clicks =
LOAD 'access_log.gz'
USING nl.basjes.pig.input.apachehttpdlog.Loader(
'%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"',
'IP:connection.client.host',
'HTTP.URI:request.firstline.uri',
'STRING:request.firstline.uri.query.foo',
'STRING:request.status.last',
'HTTP.URI:request.referer',
'STRING:request.referer.query.foo',
'HTTP.USERAGENT:request.user-agent')
AS (
ConnectionClientHost,
RequestFirstlineUri,
RequestFirstlineUriQueryFoo,
RequestStatusLast,
RequestReferer,
RequestRefererQueryFoo,
RequestUseragent);
*
>From here you can do as you want with the resulting tuples. Note that
almost everything is output as a chararray, yet things that seem like
number (based on the sloppy typing) are output as longs.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
