Re: [rsyslog] Are we building an ERK stack?
Seeing people referring to "REK" made me smile this morning. We've been referring to our setup as REK for quit awhile - http://www.meetup.com/RVA-Data-Hackers/events/214996202/ I don't really see a reason for redis in the equation unless it's already part of your infrastructure though. imkafka is potentially interesting however. I wrote imczmq for rsyslog so I have a little experience writing input plugins - if there are others with a need for a such a thing that would be a fun collaboration. Brian On Wed, Nov 23, 2016 at 6:52 AM Bob Gregory wrote: > There've been a few discussions over the last few days that are all > pointing in the same direction: > > * Is it better to use Rsyslog's omelasticsearch rather than pushing to > logstash? > * Should we have a minimal log shipper component as distinct from rsyslog's > processing capabilities? > * Ought we to have an imhiredis module? > > Really what we're talking about is replacing Logstash (and the various > beats) with rsyslog. I'm perfectly happy with that, Logstash is a > resource-expensive and fickle beast that spoils my otherwise pristine log > pipeline, but I do think the community ought to think about whether this is > the direction they want to take. > > For my part, I'm quite happy to help build an imhiredis (and imkafka?) > module but only if I can actually dogfood it, which means replacing > Logstash in our own environment. > > For that, I'd like to see better support for GeoIP tagging, a Riemann > output plugin, some better guidance on "failed message queues", etc. etc. > etc. > > Are we jointly interested in building the REK stack and, if so, can we > start to work out the feature set we're missing, and the documentation we'd > need for this to work? I'm a little concerned that if we tackle the usecase > piece-meal, we'll end up with lots of disjointed parts that don't really > solve the problem: logstash is not an adequate logstash. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] conversion functions
We currently have issues open for a few different conversion functions (some date related, ip2num) should we start down the path of having a bunch of different functions for all the permutations of source and destination formats? or should we have a convert() function that lets you specify the source and destination formats There are a bunch of 'types' that can all be represented internally with a fairly simple binary representation decimal hex (including variations like we commonly see for mac addresses and ipv6) octal ipv4 dates these may be extended slightly from being integers to having a fixed-point fractional component (not floating point, which can't properly represent 0.1) so should we create a function convert(var,sformat,dformatt[,extradata]) where the function converts the variable contents from sformat to the internal binary representation and then from that to the dformat (allowing the optional extradata parameter to specify tweaks to the dformat) thoughts. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] conversion functions
I feel it's better to have parse_(expr) and cast_(expr) family of functions that return the appropriate value. Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and cast_int(parse_double(x)) returns 12. Cast detects the source type and acts accordingly, parse expects expression to produce a string. On Nov 27, 2016 4:14 AM, "David Lang" wrote: > We currently have issues open for a few different conversion functions > (some date related, ip2num) > > should we start down the path of having a bunch of different functions for > all the permutations of source and destination formats? or should we have a > convert() function that lets you specify the source and destination formats > > There are a bunch of 'types' that can all be represented internally with a > fairly simple binary representation > > decimal > hex (including variations like we commonly see for mac addresses and ipv6) > octal > ipv4 > dates > > these may be extended slightly from being integers to having a fixed-point > fractional component (not floating point, which can't properly represent > 0.1) > > so should we create a function > > convert(var,sformat,dformatt[,extradata]) > > where the function converts the variable contents from sformat to the > internal binary representation and then from that to the dformat (allowing > the optional extradata parameter to specify tweaks to the dformat) > > thoughts. > > David Lang > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] conversion functions
On Sun, 27 Nov 2016, singh.janmejay wrote: I feel it's better to have parse_(expr) and cast_(expr) family of functions that return the appropriate value. Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and cast_int(parse_double(x)) returns 12. Cast detects the source type and acts accordingly, parse expects expression to produce a string. are you meaning to have these in the rsyslog.conf file or as functions in the C source? David Lang On Nov 27, 2016 4:14 AM, "David Lang" wrote: We currently have issues open for a few different conversion functions (some date related, ip2num) should we start down the path of having a bunch of different functions for all the permutations of source and destination formats? or should we have a convert() function that lets you specify the source and destination formats There are a bunch of 'types' that can all be represented internally with a fairly simple binary representation decimal hex (including variations like we commonly see for mac addresses and ipv6) octal ipv4 dates these may be extended slightly from being integers to having a fixed-point fractional component (not floating point, which can't properly represent 0.1) so should we create a function convert(var,sformat,dformatt[,extradata]) where the function converts the variable contents from sformat to the internal binary representation and then from that to the dformat (allowing the optional extradata parameter to specify tweaks to the dformat) thoughts. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] conversion functions
I meant rainerscript functions. It'll boil down to equivalent functions in C too, but that is implementation detail. Did I misunderstand the context? On Nov 27, 2016 10:49 AM, "David Lang" wrote: > On Sun, 27 Nov 2016, singh.janmejay wrote: > > I feel it's better to have parse_(expr) and cast_(expr) family >> of functions that return the appropriate value. >> >> Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and >> cast_int(parse_double(x)) returns 12. >> >> Cast detects the source type and acts accordingly, parse expects >> expression >> to produce a string. >> > > are you meaning to have these in the rsyslog.conf file or as functions in > the C source? > > David Lang > > On Nov 27, 2016 4:14 AM, "David Lang" wrote: >> >> We currently have issues open for a few different conversion functions >>> (some date related, ip2num) >>> >>> should we start down the path of having a bunch of different functions >>> for >>> all the permutations of source and destination formats? or should we >>> have a >>> convert() function that lets you specify the source and destination >>> formats >>> >>> There are a bunch of 'types' that can all be represented internally with >>> a >>> fairly simple binary representation >>> >>> decimal >>> hex (including variations like we commonly see for mac addresses and >>> ipv6) >>> octal >>> ipv4 >>> dates >>> >>> these may be extended slightly from being integers to having a >>> fixed-point >>> fractional component (not floating point, which can't properly represent >>> 0.1) >>> >>> so should we create a function >>> >>> convert(var,sformat,dformatt[,extradata]) >>> >>> where the function converts the variable contents from sformat to the >>> internal binary representation and then from that to the dformat >>> (allowing >>> the optional extradata parameter to specify tweaks to the dformat) >>> >>> thoughts. >>> >>> David Lang >>> ___ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] conversion functions
On Sun, 27 Nov 2016, singh.janmejay wrote:
I meant rainerscript functions. It'll boil down to equivalent functions in
C too, but that is implementation detail.
Did I misunderstand the context?
the context I was thinking of was rsyslog.conf. I just wasn't thinking of how
your suggestion would work.
so you are suggesting that we have X2num and then num2x functions, so if you
needed to convert hex to octal you would do
set $.var1 = hex2num("$.var");
set $.var2 = num2oct("$.var1);
vs
set $.var2 = convert("$.var","hex","oct");
am I understanding you correctly?
David Lang
On Nov 27, 2016 10:49 AM, "David Lang" wrote:
On Sun, 27 Nov 2016, singh.janmejay wrote:
I feel it's better to have parse_(expr) and cast_(expr) family
of functions that return the appropriate value.
Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and
cast_int(parse_double(x)) returns 12.
Cast detects the source type and acts accordingly, parse expects
expression
to produce a string.
are you meaning to have these in the rsyslog.conf file or as functions in
the C source?
David Lang
On Nov 27, 2016 4:14 AM, "David Lang" wrote:
We currently have issues open for a few different conversion functions
(some date related, ip2num)
should we start down the path of having a bunch of different functions
for
all the permutations of source and destination formats? or should we
have a
convert() function that lets you specify the source and destination
formats
There are a bunch of 'types' that can all be represented internally with
a
fairly simple binary representation
decimal
hex (including variations like we commonly see for mac addresses and
ipv6)
octal
ipv4
dates
these may be extended slightly from being integers to having a
fixed-point
fractional component (not floating point, which can't properly represent
0.1)
so should we create a function
convert(var,sformat,dformatt[,extradata])
where the function converts the variable contents from sformat to the
internal binary representation and then from that to the dformat
(allowing
the optional extradata parameter to specify tweaks to the dformat)
thoughts.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
