Re: Module for simple processing of log files
Philippe 'BooK' Bruhat wrote:
Hi,
[...]
The module works like this:
use Blah;# not a very good name for CPAN :-)
Acme::Blah! :o)
my $blah = Blah-new(
delimiter = ':',
fields= [qw( bap clank glipp plop )],
show = \show_my_data,
);
# quelques infos utiles
$blah-add_filter(
bap = 'eq zlopp',
clank = '!~ /clunk_eth/',
A hash is unordered. Um hang on, or is this a list of pairs or a hash?
);
$blah-add_file( glob *.log );
$blah-run;
When the run() method is called, the object compute a piece of Perl
code which is then eval()ed (with a localised @ARGV). The computed
code is the following (note the use of arrays for the selection pass
and of a hash for the processing):
while() {
chomp;
my @data = split qr{:};
if(( $data[1] !~ /clunk_eth/ )
( $data[0] eq zlopp ) )
This is ordered. But in any case, it's not in the same order as above.
I would like to be able to specify order in which the checks are run, so
as to be able to evaluate the check that fails 99% of the time first.
Also, I'm not sure what to suggest, but...
bap = 'eq zlopp',
...feels icky. I'd rather something that would spit out errors at
compile time, not at eval time. But I can't think of anything
approaching that level of tersity. At the very least:
bap = { 'eq' = 'zlopp' }
clunk = { '~!' = qr/^foom\d+$/ }
sput = { 'between' = [10, 100] }
Hmm, how do I add a new conditional widget, like 'between' above? e.g. I
want to extract all HTTP transactions whose lengths are prime fibonacci
numbers.
David
Re: Module for simple processing of log files
Le mardi 29 mars 2005 à 19:05, David Landgren écrivait:
$blah-add_filter(
bap = 'eq zlopp',
clank = '!~ /clunk_eth/',
A hash is unordered. Um hang on, or is this a list of pairs or a hash?
It is a list of pairs. For the moment, the right part is pushed on an
array reference stored in a hash indexed by the left part. So yes, I
can do $blah-add_filter( bap = 'ne zlopp', bap = 'ne klonk' )
chomp;
my @data = split qr{:};
if(( $data[1] !~ /clunk_eth/ )
( $data[0] eq zlopp ) )
This is ordered. But in any case, it's not in the same order as above.
Sure, it's a hash.
I would like to be able to specify order in which the checks are run, so
as to be able to evaluate the check that fails 99% of the time first.
Good point.
Also, I'm not sure what to suggest, but...
bap = 'eq zlopp',
...feels icky. I'd rather something that would spit out errors at
compile time, not at eval time. But I can't think of anything
approaching that level of tersity. At the very least:
bap = { 'eq' = 'zlopp' }
clunk = { '~!' = qr/^foom\d+$/ }
sput = { 'between' = [10, 100] }
Hmm, how do I add a new conditional widget, like 'between' above? e.g. I
want to extract all HTTP transactions whose lengths are prime fibonacci
numbers.
Moving away from the inner Perl code is a very good option, which Yves
mentionned as well. My first version simply puts the strings into $code,
which has a lot of red flashing lights all around.
Does your asking this kind of questions mean the module is interesting
enough to be posted on CPAN?
--
Philippe BooK Bruhat
When you deal in weapons, there are no winners... only losers.
(Moral from Groo The Wanderer #31 (Epic))
Re: Module for simple processing of log files
Le mardi 29 mars 2005 à 17:52, Orton, Yves écrivait: Any other name ideas or comments about the module and its interface? I started working on a project like this but never got around to finishing it. I called it Generic Record Processing System IE GRPS. The point being that this isnt a facility related to parsing log files, its a facility relating to processing any file of parsable records in a mechanical way. This is one reason why I didn't like the Log:: prefix. This also implies that my Regexp::Log suite of modules is badly named as well. Interesting rules would be stuff like set membership is field x in set Y (which would be implemented as a hash lookup etc). Also interesting would be a framwork for specifiying which ruleset to apply based on filename/directory conventions. Additionally stuff like record transformation, prefix matching and logical evaluation would be cool too. Set operations? Good idea. If the field is a string and the set is a set of strings, computing a regular expression with Regexp::Assemble might be a good option as well. And then, caching the generated code (e.g. because Regexp::Assemble takes some time to run) could be a bonus. -- Philippe BooK Bruhat For every winner, there must be one or more losers. (Moral to the Sage story in Groo #111 (Epic))
Re: Should DSLIP codes be updated?
* Robert Rothenberg [EMAIL PROTECTED] [2005-03-29T14:16:11] Some food for thought and debate. I'm wondering if the DSLIP codes [1] be updated, if revamped altogether. Note the following issues: I vote for eliminated. -- rjbs pgpxUKHHyPWvh.pgp Description: PGP signature
Re: Should DSLIP codes be updated?
On Tue, Mar 29, 2005 at 03:06:33PM -0600, Andy Lester wrote: On Tue, Mar 29, 2005 at 07:16:11PM +, Robert Rothenberg ([EMAIL PROTECTED]) wrote: Some food for thought and debate. I'm wondering if the DSLIP codes [1] be updated, if revamped altogether. Note the following issues: Or thrown away entirely, along with the rest of the the archaic idea of a module list. The Module List is dead. Module Registration is different. Tim.
Re: Should DSLIP codes be updated?
On Tue, Mar 29, 2005 at 11:06:37PM +0100, Tim Bunce ([EMAIL PROTECTED]) wrote: Or thrown away entirely, along with the rest of the the archaic idea of a module list. The Module List is dead. Module Registration is different. Mea culpa. I'll rephrase. Or thrown away entirely, along with the rest of the archaic idea of module registration. The time has come to recognize that CPAN is an unregulated free-for-all, and that the existing way of trying to wrap our heads around its contents hasn't scaled and needs to go away. The good parts (knowing who is authoritative for a module) need to get pulled out, and put into a new system. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Should DSLIP codes be updated?
On Tue, Mar 29, 2005 at 04:14:46PM -0600, Andy Lester wrote: On Tue, Mar 29, 2005 at 11:06:37PM +0100, Tim Bunce ([EMAIL PROTECTED]) wrote: Or thrown away entirely, along with the rest of the the archaic idea of a module list. The Module List is dead. Module Registration is different. Mea culpa. I'll rephrase. Or thrown away entirely, along with the rest of the archaic idea of module registration. :-) The time has come to recognize that CPAN is an unregulated free-for-all, and that the existing way of trying to wrap our heads around its contents hasn't scaled and needs to go away. The good parts (knowing who is authoritative for a module) need to get pulled out, and put into a new system. I don't mind if the current system gets fixed (which could be done, per my previous emails) or something new gets implemented. Ultimately what matters most is that something gets done by someone. Personally I've done my time, all ten years of it, as a please give your modules a sensible name advocate. I'm letting others do that now, to whatever extent they want. Tim.
Re: Should DSLIP codes be updated?
On 29/03/2005 22:14 Andy Lester wrote: Mea culpa. I'll rephrase. Or thrown away entirely, along with the rest of the archaic idea of module registration. The time has come to recognize that CPAN is an unregulated free-for-all, and that the existing way of trying to wrap our heads around its contents hasn't scaled and needs to go away. The good parts (knowing who is authoritative for a module) need to get pulled out, and put into a new system. I'm sympathetic to the idea, but some of the information in DSLIP is useful and shouldn't be thrown away (such as how supported, alpha/beta/mature, and license). What isn't in META.yml should go there. Other things such as interface style could be determined automatically, though it may take hints from developers. I'm not sure if it really matters if a module uses embedded Java, Perl6, Lisp, or whatever so long as it works. What's more important is how to indicate what external programs or libraries a module uses. (As a CPAN Tester this is the biggest bugbear!) Support information is a little more nuanced, and I'm not sure how that should be handled. Rob
Re: Should DSLIP codes be updated?
On Tue, Mar 29, 2005 at 11:03:09PM +, Robert Rothenberg wrote: I'm sympathetic to the idea, but some of the information in DSLIP is useful and shouldn't be thrown away (such as how supported, alpha/beta/mature, and license). What isn't in META.yml should go there. I'm much less interested in whether the author thinks the work is mature, than what the users thinks. We already have a mechanism to create version numbers that indicate a developer release. For me, having a development/stable indicator than an alpha/beta/mature label. How many times have you found a module with a version less than '0.10' that was actually stable mature? On the other hand, sometimes 'mature' modules get overhauled, abandoned or forked and aren't really 'stable'. Mark -- http://mark.stosberg.com/
Re: Module for simple processing of log files
Le mardi 29 mars 2005 à 17:52, Orton, Yves écrivait: I started working on a project like this but never got around to finishing it. I called it Generic Record Processing System IE GRPS. The point being that this isnt a facility related to parsing log files, its a facility relating to processing any file of parsable records in a mechanical way. Then what do you think of Record::Processor? -- Philippe BooK Bruhat You never know what love is until you lose it. (Moral from Groo The Wanderer #38 (Epic))
