Re: [rsyslog] ommongodb saves integer values as string
any feedback on this? I'm using the latest release 8.6.0.
On 15/12/14 10:51, proinity GmbH wrote:
numeric values are not stored as integer values in the mongodb.
e.g. "bytes":"123" instead of "bytes":123
I tried it with the following template but had no success:
template(name="tpl1" type="list") {
constant(value="\"bytes\":")
property(name="$!bytes" type="int")
constant(value=", \"somestring\":\"")
property(name="$!somestring")
constant(value="\"")
}
action(type="ommongodb" server="127.0.0.1" db="test" collection="logs"
uid="test" pwd="" template="tpl1")
when I save it to a file (omfile) it makes it correctly (without the
quotes): "bytes":123
is this a known limitation of ommongodb that it stores all values as
strings? do I need to define the data type somewhere explicitly?
TIA
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards wrote: > So here it is: > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > I have deliberately left out the impstats calls as I saw you included them > in your source, so feel free to add them again ;) > > Note that there is a different approach to parameters: my version has only > very few hardcoded ones, but instead permits the user to set actual kafka > parameters via the confparam and topicparam parameters. In my experience, > this is far better with a rapidly evolving backend, because now any new or > changed parameter is immediately available. > > I'll write up some bare doc later today. So far this is tested on Ubuntu > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > could test it - especially as I have only very limited Kafka know-how and > no real use case myself. > > Thanks, > Rainer > Great! Looks much better than my original version, the "dynamic" configuration for conf and topicconf are definitely a win there (hardcoding is the worst). I had submitted my pull request with the template enabled topics, I had to shuffle some of the items around to make the topics work on each call of writeKafka. Found here: https://github.com/rsyslog/rsyslog/pull/186 Tait ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
2014-12-15 13:39 GMT+01:00 Rainer Gerhards : > 2014-12-15 13:34 GMT+01:00 Brian Knox : > >> So 8.7 is aiming for Jan 13th? > > > yup, the next date is now always given in the "current version" box on the > rsyslog homepage. > > We dediced to do an release every 6 weeks, and I will try to stick to that. > > >> I'll try to get some zeromq input and >> output improvements done for that release then as well. >> > > That would be great! > important side-note: given the experience with 8.6.0 where I got very busy to make the date due to some problems with a recent new feature, I'll probably freeze a week before the actual release and use that week to have packages created and such. Just so that you know. Rainer > > Rainer > >> >> Cheers, and enjoy your vacation! >> Brian >> >> On Mon, Dec 15, 2014 at 6:54 AM, Rainer Gerhards < >> rgerha...@hq.adiscon.com> >> wrote: >> > >> > 2014-12-15 12:29 GMT+01:00 Brian Knox : >> > >> > > This is great, because I was thinking I should start on an output >> plugin >> > > for kafka, and now I can be lazy ;) I'll be glad to test this module >> > > against our kafka infrastructure at work! >> > > >> > > >> > excellent! Please note that I will leave Friday for a holiday vacation, >> and >> > I probably won't touch the code when away. IAW: early bug reports are >> > appreciated. I still hope to get this module in in 8.7.0 on Jan, 13th. >> > >> > Rainer >> > >> > >> > > Brian >> > > >> > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards < >> > rgerha...@hq.adiscon.com >> > > > >> > > wrote: >> > > > >> > > > So here it is: >> > > > >> > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka >> > > > >> > > > I have deliberately left out the impstats calls as I saw you >> included >> > > them >> > > > in your source, so feel free to add them again ;) >> > > > >> > > > Note that there is a different approach to parameters: my version >> has >> > > only >> > > > very few hardcoded ones, but instead permits the user to set actual >> > kafka >> > > > parameters via the confparam and topicparam parameters. In my >> > experience, >> > > > this is far better with a rapidly evolving backend, because now any >> new >> > > or >> > > > changed parameter is immediately available. >> > > > >> > > > I'll write up some bare doc later today. So far this is tested on >> > Ubuntu >> > > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone >> else) >> > > > could test it - especially as I have only very limited Kafka >> know-how >> > and >> > > > no real use case myself. >> > > > >> > > > Thanks, >> > > > Rainer >> > > > >> > > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : >> > > > >> > > > > Fantastic, that's one area I definitely omitted. I look forward to >> > > > > contributing to this great project! >> > > > > >> > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards >> > > > > wrote: >> > > > > > excellent! I hope I have the initial commit tomorrow morning. >> Now >> > > > > working a >> > > > > > bit on error detection... >> > > > > > >> > > > > > Rainer >> > > > > > >> > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : >> > > > > > >> > > > > >> I think your idea works; if you commit yours (which is way more >> > > robust >> > > > > >> than mine), I can submit a pull request for the additional >> items >> > my >> > > > > >> module had that are not present in yours. >> > > > > >> >> > > > > >> The only parts of the original contribution that I re-used were >> > > parts >> > > > > >> of the send_message function - I added much more on top of it >> so >> > > that >> > > > > >> should cover all of the items I will/might add. >> > > > > >> >> > > > > >> Tait >> > > > > >> >> > > > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards >> > > > > >> wrote: >> > > > > >> > side-note on how to carry on: I have seen check the link to >> the >> > > > > original >> > > > > >> > contribution (I admit I totally overlooked it at that time). >> It >> > > did >> > > > > not >> > > > > >> > have an explicit license, which is a bit problematic. Some >> folks >> > > > would >> > > > > >> say >> > > > > >> > this means it default back to the project license, which then >> > > would >> > > > > mean >> > > > > >> > GPLv3, which would cause problems with our move to ASL 2.0. >> > > > > >> > >> > > > > >> > Here is what I have on my mind on how to solve and be able to >> > > > merge: I >> > > > > >> hope >> > > > > >> > I can commit tomorrow. I could then >> > > > > >> > >> > > > > >> > 1. pick from your code what is missing in mine >> > > > > >> > 2. apply it under your your commit --author >> > > > > >> > 3. and finally update to copyright in the header comment to >> > > reflect >> > > > > both >> > > > > >> us >> > > > > >> > >> > > > > >> > Would that work for you? In step 1 there is a problem if I >> run >> > > into >> > > > > code >> > > > > >> > that stems back to the original contribution, which, if so, I >> > can >> > > > not >> > > > > >> merge >> > > > > >> > do to licensing.
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
2014-12-15 13:34 GMT+01:00 Brian Knox : > So 8.7 is aiming for Jan 13th? yup, the next date is now always given in the "current version" box on the rsyslog homepage. We dediced to do an release every 6 weeks, and I will try to stick to that. > I'll try to get some zeromq input and > output improvements done for that release then as well. > That would be great! Rainer > > Cheers, and enjoy your vacation! > Brian > > On Mon, Dec 15, 2014 at 6:54 AM, Rainer Gerhards > > wrote: > > > > 2014-12-15 12:29 GMT+01:00 Brian Knox : > > > > > This is great, because I was thinking I should start on an output > plugin > > > for kafka, and now I can be lazy ;) I'll be glad to test this module > > > against our kafka infrastructure at work! > > > > > > > > excellent! Please note that I will leave Friday for a holiday vacation, > and > > I probably won't touch the code when away. IAW: early bug reports are > > appreciated. I still hope to get this module in in 8.7.0 on Jan, 13th. > > > > Rainer > > > > > > > Brian > > > > > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards < > > rgerha...@hq.adiscon.com > > > > > > > wrote: > > > > > > > > So here it is: > > > > > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > > > > > I have deliberately left out the impstats calls as I saw you included > > > them > > > > in your source, so feel free to add them again ;) > > > > > > > > Note that there is a different approach to parameters: my version has > > > only > > > > very few hardcoded ones, but instead permits the user to set actual > > kafka > > > > parameters via the confparam and topicparam parameters. In my > > experience, > > > > this is far better with a rapidly evolving backend, because now any > new > > > or > > > > changed parameter is immediately available. > > > > > > > > I'll write up some bare doc later today. So far this is tested on > > Ubuntu > > > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone > else) > > > > could test it - especially as I have only very limited Kafka know-how > > and > > > > no real use case myself. > > > > > > > > Thanks, > > > > Rainer > > > > > > > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > > > > > > > Fantastic, that's one area I definitely omitted. I look forward to > > > > > contributing to this great project! > > > > > > > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > > > > wrote: > > > > > > excellent! I hope I have the initial commit tomorrow morning. Now > > > > > working a > > > > > > bit on error detection... > > > > > > > > > > > > Rainer > > > > > > > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > > > > > > > >> I think your idea works; if you commit yours (which is way more > > > robust > > > > > >> than mine), I can submit a pull request for the additional items > > my > > > > > >> module had that are not present in yours. > > > > > >> > > > > > >> The only parts of the original contribution that I re-used were > > > parts > > > > > >> of the send_message function - I added much more on top of it so > > > that > > > > > >> should cover all of the items I will/might add. > > > > > >> > > > > > >> Tait > > > > > >> > > > > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > > > > >> wrote: > > > > > >> > side-note on how to carry on: I have seen check the link to > the > > > > > original > > > > > >> > contribution (I admit I totally overlooked it at that time). > It > > > did > > > > > not > > > > > >> > have an explicit license, which is a bit problematic. Some > folks > > > > would > > > > > >> say > > > > > >> > this means it default back to the project license, which then > > > would > > > > > mean > > > > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > > > > >> > > > > > > >> > Here is what I have on my mind on how to solve and be able to > > > > merge: I > > > > > >> hope > > > > > >> > I can commit tomorrow. I could then > > > > > >> > > > > > > >> > 1. pick from your code what is missing in mine > > > > > >> > 2. apply it under your your commit --author > > > > > >> > 3. and finally update to copyright in the header comment to > > > reflect > > > > > both > > > > > >> us > > > > > >> > > > > > > >> > Would that work for you? In step 1 there is a problem if I run > > > into > > > > > code > > > > > >> > that stems back to the original contribution, which, if so, I > > can > > > > not > > > > > >> merge > > > > > >> > do to licensing. I hope that won't be the case. > > > > > >> > > > > > > >> > If you like, you can of course create a PR based on my commit > > > > > tomorrow, > > > > > >> you > > > > > >> > just need to make sure that all code that is added is > originally > > > > > written > > > > > >> by > > > > > >> > you or under some license that is OK with ASL 2.0 (like the > BSD > > > > > 2-clause > > > > > >> > from librdkafka). > > > > > >> > > > > > > >> > What would work for you? > > > > > >> > > > > > > >> > Rainer > > > > > >> > > > > > > >>
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
So 8.7 is aiming for Jan 13th? I'll try to get some zeromq input and output improvements done for that release then as well. Cheers, and enjoy your vacation! Brian On Mon, Dec 15, 2014 at 6:54 AM, Rainer Gerhards wrote: > > 2014-12-15 12:29 GMT+01:00 Brian Knox : > > > This is great, because I was thinking I should start on an output plugin > > for kafka, and now I can be lazy ;) I'll be glad to test this module > > against our kafka infrastructure at work! > > > > > excellent! Please note that I will leave Friday for a holiday vacation, and > I probably won't touch the code when away. IAW: early bug reports are > appreciated. I still hope to get this module in in 8.7.0 on Jan, 13th. > > Rainer > > > > Brian > > > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards < > rgerha...@hq.adiscon.com > > > > > wrote: > > > > > > So here it is: > > > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > > > I have deliberately left out the impstats calls as I saw you included > > them > > > in your source, so feel free to add them again ;) > > > > > > Note that there is a different approach to parameters: my version has > > only > > > very few hardcoded ones, but instead permits the user to set actual > kafka > > > parameters via the confparam and topicparam parameters. In my > experience, > > > this is far better with a rapidly evolving backend, because now any new > > or > > > changed parameter is immediately available. > > > > > > I'll write up some bare doc later today. So far this is tested on > Ubuntu > > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > > > could test it - especially as I have only very limited Kafka know-how > and > > > no real use case myself. > > > > > > Thanks, > > > Rainer > > > > > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > > > > > Fantastic, that's one area I definitely omitted. I look forward to > > > > contributing to this great project! > > > > > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > > > wrote: > > > > > excellent! I hope I have the initial commit tomorrow morning. Now > > > > working a > > > > > bit on error detection... > > > > > > > > > > Rainer > > > > > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > > > > > >> I think your idea works; if you commit yours (which is way more > > robust > > > > >> than mine), I can submit a pull request for the additional items > my > > > > >> module had that are not present in yours. > > > > >> > > > > >> The only parts of the original contribution that I re-used were > > parts > > > > >> of the send_message function - I added much more on top of it so > > that > > > > >> should cover all of the items I will/might add. > > > > >> > > > > >> Tait > > > > >> > > > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > > > >> wrote: > > > > >> > side-note on how to carry on: I have seen check the link to the > > > > original > > > > >> > contribution (I admit I totally overlooked it at that time). It > > did > > > > not > > > > >> > have an explicit license, which is a bit problematic. Some folks > > > would > > > > >> say > > > > >> > this means it default back to the project license, which then > > would > > > > mean > > > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > > > >> > > > > > >> > Here is what I have on my mind on how to solve and be able to > > > merge: I > > > > >> hope > > > > >> > I can commit tomorrow. I could then > > > > >> > > > > > >> > 1. pick from your code what is missing in mine > > > > >> > 2. apply it under your your commit --author > > > > >> > 3. and finally update to copyright in the header comment to > > reflect > > > > both > > > > >> us > > > > >> > > > > > >> > Would that work for you? In step 1 there is a problem if I run > > into > > > > code > > > > >> > that stems back to the original contribution, which, if so, I > can > > > not > > > > >> merge > > > > >> > do to licensing. I hope that won't be the case. > > > > >> > > > > > >> > If you like, you can of course create a PR based on my commit > > > > tomorrow, > > > > >> you > > > > >> > just need to make sure that all code that is added is originally > > > > written > > > > >> by > > > > >> > you or under some license that is OK with ASL 2.0 (like the BSD > > > > 2-clause > > > > >> > from librdkafka). > > > > >> > > > > > >> > What would work for you? > > > > >> > > > > > >> > Rainer > > > > >> > > > > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards < > > > rgerha...@hq.adiscon.com > > > > >: > > > > >> > > > > > >> >> just sent it your way :-) > > > > >> >> > > > > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > > > > >> >> > > > > >> >>> > just leave it as is for now. I need to merge manually in any > > > > case, as > > > > >> >>> both > > > > >> >>> > commits will be just a module. Bear a day with me, I don't > > want > > > to > > > > >> do a > > > > >> >>> > hasty commit of my unfinished work. If you like, I can > share a > > > >
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
2014-12-15 12:29 GMT+01:00 Brian Knox : > This is great, because I was thinking I should start on an output plugin > for kafka, and now I can be lazy ;) I'll be glad to test this module > against our kafka infrastructure at work! > > excellent! Please note that I will leave Friday for a holiday vacation, and I probably won't touch the code when away. IAW: early bug reports are appreciated. I still hope to get this module in in 8.7.0 on Jan, 13th. Rainer > Brian > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards > > wrote: > > > > So here it is: > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > I have deliberately left out the impstats calls as I saw you included > them > > in your source, so feel free to add them again ;) > > > > Note that there is a different approach to parameters: my version has > only > > very few hardcoded ones, but instead permits the user to set actual kafka > > parameters via the confparam and topicparam parameters. In my experience, > > this is far better with a rapidly evolving backend, because now any new > or > > changed parameter is immediately available. > > > > I'll write up some bare doc later today. So far this is tested on Ubuntu > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > > could test it - especially as I have only very limited Kafka know-how and > > no real use case myself. > > > > Thanks, > > Rainer > > > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > > > Fantastic, that's one area I definitely omitted. I look forward to > > > contributing to this great project! > > > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > > wrote: > > > > excellent! I hope I have the initial commit tomorrow morning. Now > > > working a > > > > bit on error detection... > > > > > > > > Rainer > > > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > > > >> I think your idea works; if you commit yours (which is way more > robust > > > >> than mine), I can submit a pull request for the additional items my > > > >> module had that are not present in yours. > > > >> > > > >> The only parts of the original contribution that I re-used were > parts > > > >> of the send_message function - I added much more on top of it so > that > > > >> should cover all of the items I will/might add. > > > >> > > > >> Tait > > > >> > > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > > >> wrote: > > > >> > side-note on how to carry on: I have seen check the link to the > > > original > > > >> > contribution (I admit I totally overlooked it at that time). It > did > > > not > > > >> > have an explicit license, which is a bit problematic. Some folks > > would > > > >> say > > > >> > this means it default back to the project license, which then > would > > > mean > > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > > >> > > > > >> > Here is what I have on my mind on how to solve and be able to > > merge: I > > > >> hope > > > >> > I can commit tomorrow. I could then > > > >> > > > > >> > 1. pick from your code what is missing in mine > > > >> > 2. apply it under your your commit --author > > > >> > 3. and finally update to copyright in the header comment to > reflect > > > both > > > >> us > > > >> > > > > >> > Would that work for you? In step 1 there is a problem if I run > into > > > code > > > >> > that stems back to the original contribution, which, if so, I can > > not > > > >> merge > > > >> > do to licensing. I hope that won't be the case. > > > >> > > > > >> > If you like, you can of course create a PR based on my commit > > > tomorrow, > > > >> you > > > >> > just need to make sure that all code that is added is originally > > > written > > > >> by > > > >> > you or under some license that is OK with ASL 2.0 (like the BSD > > > 2-clause > > > >> > from librdkafka). > > > >> > > > > >> > What would work for you? > > > >> > > > > >> > Rainer > > > >> > > > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards < > > rgerha...@hq.adiscon.com > > > >: > > > >> > > > > >> >> just sent it your way :-) > > > >> >> > > > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > > > >> >> > > > >> >>> > just leave it as is for now. I need to merge manually in any > > > case, as > > > >> >>> both > > > >> >>> > commits will be just a module. Bear a day with me, I don't > want > > to > > > >> do a > > > >> >>> > hasty commit of my unfinished work. If you like, I can share a > > > >> working > > > >> >>> > copy, though. Just let me know. > > > >> >>> > > > > >> >>> > Rainer > > > >> >>> > > > >> >>> That sounds great, when you get a chance I'd like to take a > look. > > > I'm > > > >> >>> in no rush as well so take your time :) > > > >> >>> > > > >> >>> Tait > > > >> >>> ___ > > > >> >>> rsyslog mailing list > > > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> >>> http://www.rsyslog.com/professional-services/ > > > >> >>> What's up with rsyslog? Foll
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
This is great, because I was thinking I should start on an output plugin for kafka, and now I can be lazy ;) I'll be glad to test this module against our kafka infrastructure at work! Brian On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards wrote: > > So here it is: > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > I have deliberately left out the impstats calls as I saw you included them > in your source, so feel free to add them again ;) > > Note that there is a different approach to parameters: my version has only > very few hardcoded ones, but instead permits the user to set actual kafka > parameters via the confparam and topicparam parameters. In my experience, > this is far better with a rapidly evolving backend, because now any new or > changed parameter is immediately available. > > I'll write up some bare doc later today. So far this is tested on Ubuntu > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > could test it - especially as I have only very limited Kafka know-how and > no real use case myself. > > Thanks, > Rainer > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > Fantastic, that's one area I definitely omitted. I look forward to > > contributing to this great project! > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > wrote: > > > excellent! I hope I have the initial commit tomorrow morning. Now > > working a > > > bit on error detection... > > > > > > Rainer > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > >> I think your idea works; if you commit yours (which is way more robust > > >> than mine), I can submit a pull request for the additional items my > > >> module had that are not present in yours. > > >> > > >> The only parts of the original contribution that I re-used were parts > > >> of the send_message function - I added much more on top of it so that > > >> should cover all of the items I will/might add. > > >> > > >> Tait > > >> > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > >> wrote: > > >> > side-note on how to carry on: I have seen check the link to the > > original > > >> > contribution (I admit I totally overlooked it at that time). It did > > not > > >> > have an explicit license, which is a bit problematic. Some folks > would > > >> say > > >> > this means it default back to the project license, which then would > > mean > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > >> > > > >> > Here is what I have on my mind on how to solve and be able to > merge: I > > >> hope > > >> > I can commit tomorrow. I could then > > >> > > > >> > 1. pick from your code what is missing in mine > > >> > 2. apply it under your your commit --author > > >> > 3. and finally update to copyright in the header comment to reflect > > both > > >> us > > >> > > > >> > Would that work for you? In step 1 there is a problem if I run into > > code > > >> > that stems back to the original contribution, which, if so, I can > not > > >> merge > > >> > do to licensing. I hope that won't be the case. > > >> > > > >> > If you like, you can of course create a PR based on my commit > > tomorrow, > > >> you > > >> > just need to make sure that all code that is added is originally > > written > > >> by > > >> > you or under some license that is OK with ASL 2.0 (like the BSD > > 2-clause > > >> > from librdkafka). > > >> > > > >> > What would work for you? > > >> > > > >> > Rainer > > >> > > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards < > rgerha...@hq.adiscon.com > > >: > > >> > > > >> >> just sent it your way :-) > > >> >> > > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > > >> >> > > >> >>> > just leave it as is for now. I need to merge manually in any > > case, as > > >> >>> both > > >> >>> > commits will be just a module. Bear a day with me, I don't want > to > > >> do a > > >> >>> > hasty commit of my unfinished work. If you like, I can share a > > >> working > > >> >>> > copy, though. Just let me know. > > >> >>> > > > >> >>> > Rainer > > >> >>> > > >> >>> That sounds great, when you get a chance I'd like to take a look. > > I'm > > >> >>> in no rush as well so take your time :) > > >> >>> > > >> >>> Tait > > >> >>> ___ > > >> >>> rsyslog mailing list > > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> >>> http://www.rsyslog.com/professional-services/ > > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >> myriad > > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > >> >>> DON'T LIKE THAT. > > >> >>> > > >> >> > > >> >> > > >> > ___ > > >> > rsyslog mailing list > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > http://www.rsyslog.com/professional-services/ > > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > NOTE
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
2014-12-15 11:41 GMT+01:00 Tim Smith : > Eagerly awaiting a new build or build instructions - I can throw real data > streams and hardware for testing :) > > which platform do you use? Rainer > Mon, Dec 15, 2014 at 3:34 PM, Rainer Gerhards > wrote: > > > > So here it is: > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > I have deliberately left out the impstats calls as I saw you included > them > > in your source, so feel free to add them again ;) > > > > Note that there is a different approach to parameters: my version has > only > > very few hardcoded ones, but instead permits the user to set actual kafka > > parameters via the confparam and topicparam parameters. In my experience, > > this is far better with a rapidly evolving backend, because now any new > or > > changed parameter is immediately available. > > > > I'll write up some bare doc later today. So far this is tested on Ubuntu > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > > could test it - especially as I have only very limited Kafka know-how and > > no real use case myself. > > > > Thanks, > > Rainer > > > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > > > Fantastic, that's one area I definitely omitted. I look forward to > > > contributing to this great project! > > > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > > wrote: > > > > excellent! I hope I have the initial commit tomorrow morning. Now > > > working a > > > > bit on error detection... > > > > > > > > Rainer > > > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > > > >> I think your idea works; if you commit yours (which is way more > robust > > > >> than mine), I can submit a pull request for the additional items my > > > >> module had that are not present in yours. > > > >> > > > >> The only parts of the original contribution that I re-used were > parts > > > >> of the send_message function - I added much more on top of it so > that > > > >> should cover all of the items I will/might add. > > > >> > > > >> Tait > > > >> > > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > > >> wrote: > > > >> > side-note on how to carry on: I have seen check the link to the > > > original > > > >> > contribution (I admit I totally overlooked it at that time). It > did > > > not > > > >> > have an explicit license, which is a bit problematic. Some folks > > would > > > >> say > > > >> > this means it default back to the project license, which then > would > > > mean > > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > > >> > > > > >> > Here is what I have on my mind on how to solve and be able to > > merge: I > > > >> hope > > > >> > I can commit tomorrow. I could then > > > >> > > > > >> > 1. pick from your code what is missing in mine > > > >> > 2. apply it under your your commit --author > > > >> > 3. and finally update to copyright in the header comment to > reflect > > > both > > > >> us > > > >> > > > > >> > Would that work for you? In step 1 there is a problem if I run > into > > > code > > > >> > that stems back to the original contribution, which, if so, I can > > not > > > >> merge > > > >> > do to licensing. I hope that won't be the case. > > > >> > > > > >> > If you like, you can of course create a PR based on my commit > > > tomorrow, > > > >> you > > > >> > just need to make sure that all code that is added is originally > > > written > > > >> by > > > >> > you or under some license that is OK with ASL 2.0 (like the BSD > > > 2-clause > > > >> > from librdkafka). > > > >> > > > > >> > What would work for you? > > > >> > > > > >> > Rainer > > > >> > > > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards < > > rgerha...@hq.adiscon.com > > > >: > > > >> > > > > >> >> just sent it your way :-) > > > >> >> > > > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > > > >> >> > > > >> >>> > just leave it as is for now. I need to merge manually in any > > > case, as > > > >> >>> both > > > >> >>> > commits will be just a module. Bear a day with me, I don't > want > > to > > > >> do a > > > >> >>> > hasty commit of my unfinished work. If you like, I can share a > > > >> working > > > >> >>> > copy, though. Just let me know. > > > >> >>> > > > > >> >>> > Rainer > > > >> >>> > > > >> >>> That sounds great, when you get a chance I'd like to take a > look. > > > I'm > > > >> >>> in no rush as well so take your time :) > > > >> >>> > > > >> >>> Tait > > > >> >>> ___ > > > >> >>> rsyslog mailing list > > > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> >>> http://www.rsyslog.com/professional-services/ > > > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > > > >> myriad > > > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > > > you > > > >> >>> DON'T LIKE THAT. > > > >> >>> > > > >> >> > > >
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
Eagerly awaiting a new build or build instructions - I can throw real data streams and hardware for testing :) On Mon, Dec 15, 2014 at 3:34 PM, Rainer Gerhards wrote: > > So here it is: > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > I have deliberately left out the impstats calls as I saw you included them > in your source, so feel free to add them again ;) > > Note that there is a different approach to parameters: my version has only > very few hardcoded ones, but instead permits the user to set actual kafka > parameters via the confparam and topicparam parameters. In my experience, > this is far better with a rapidly evolving backend, because now any new or > changed parameter is immediately available. > > I'll write up some bare doc later today. So far this is tested on Ubuntu > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) > could test it - especially as I have only very limited Kafka know-how and > no real use case myself. > > Thanks, > Rainer > > 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > > > Fantastic, that's one area I definitely omitted. I look forward to > > contributing to this great project! > > > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > > wrote: > > > excellent! I hope I have the initial commit tomorrow morning. Now > > working a > > > bit on error detection... > > > > > > Rainer > > > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > > > >> I think your idea works; if you commit yours (which is way more robust > > >> than mine), I can submit a pull request for the additional items my > > >> module had that are not present in yours. > > >> > > >> The only parts of the original contribution that I re-used were parts > > >> of the send_message function - I added much more on top of it so that > > >> should cover all of the items I will/might add. > > >> > > >> Tait > > >> > > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > > >> wrote: > > >> > side-note on how to carry on: I have seen check the link to the > > original > > >> > contribution (I admit I totally overlooked it at that time). It did > > not > > >> > have an explicit license, which is a bit problematic. Some folks > would > > >> say > > >> > this means it default back to the project license, which then would > > mean > > >> > GPLv3, which would cause problems with our move to ASL 2.0. > > >> > > > >> > Here is what I have on my mind on how to solve and be able to > merge: I > > >> hope > > >> > I can commit tomorrow. I could then > > >> > > > >> > 1. pick from your code what is missing in mine > > >> > 2. apply it under your your commit --author > > >> > 3. and finally update to copyright in the header comment to reflect > > both > > >> us > > >> > > > >> > Would that work for you? In step 1 there is a problem if I run into > > code > > >> > that stems back to the original contribution, which, if so, I can > not > > >> merge > > >> > do to licensing. I hope that won't be the case. > > >> > > > >> > If you like, you can of course create a PR based on my commit > > tomorrow, > > >> you > > >> > just need to make sure that all code that is added is originally > > written > > >> by > > >> > you or under some license that is OK with ASL 2.0 (like the BSD > > 2-clause > > >> > from librdkafka). > > >> > > > >> > What would work for you? > > >> > > > >> > Rainer > > >> > > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards < > rgerha...@hq.adiscon.com > > >: > > >> > > > >> >> just sent it your way :-) > > >> >> > > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > > >> >> > > >> >>> > just leave it as is for now. I need to merge manually in any > > case, as > > >> >>> both > > >> >>> > commits will be just a module. Bear a day with me, I don't want > to > > >> do a > > >> >>> > hasty commit of my unfinished work. If you like, I can share a > > >> working > > >> >>> > copy, though. Just let me know. > > >> >>> > > > >> >>> > Rainer > > >> >>> > > >> >>> That sounds great, when you get a chance I'd like to take a look. > > I'm > > >> >>> in no rush as well so take your time :) > > >> >>> > > >> >>> Tait > > >> >>> ___ > > >> >>> rsyslog mailing list > > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> >>> http://www.rsyslog.com/professional-services/ > > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >> myriad > > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > >> >>> DON'T LIKE THAT. > > >> >>> > > >> >> > > >> >> > > >> > ___ > > >> > rsyslog mailing list > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > http://www.rsyslog.com/professional-services/ > > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>
Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)
So here it is: https://github.com/rsyslog/rsyslog/tree/master-omkafka I have deliberately left out the impstats calls as I saw you included them in your source, so feel free to add them again ;) Note that there is a different approach to parameters: my version has only very few hardcoded ones, but instead permits the user to set actual kafka parameters via the confparam and topicparam parameters. In my experience, this is far better with a rapidly evolving backend, because now any new or changed parameter is immediately available. I'll write up some bare doc later today. So far this is tested on Ubuntu 14.04LTS only, and I would *deeply* appreciate if you (and anyone else) could test it - especially as I have only very limited Kafka know-how and no real use case myself. Thanks, Rainer 2014-12-15 0:07 GMT+01:00 Tait Clarridge : > Fantastic, that's one area I definitely omitted. I look forward to > contributing to this great project! > > On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards > wrote: > > excellent! I hope I have the initial commit tomorrow morning. Now > working a > > bit on error detection... > > > > Rainer > > > > 2014-12-14 19:14 GMT+01:00 Tait Clarridge : > > > >> I think your idea works; if you commit yours (which is way more robust > >> than mine), I can submit a pull request for the additional items my > >> module had that are not present in yours. > >> > >> The only parts of the original contribution that I re-used were parts > >> of the send_message function - I added much more on top of it so that > >> should cover all of the items I will/might add. > >> > >> Tait > >> > >> On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards > >> wrote: > >> > side-note on how to carry on: I have seen check the link to the > original > >> > contribution (I admit I totally overlooked it at that time). It did > not > >> > have an explicit license, which is a bit problematic. Some folks would > >> say > >> > this means it default back to the project license, which then would > mean > >> > GPLv3, which would cause problems with our move to ASL 2.0. > >> > > >> > Here is what I have on my mind on how to solve and be able to merge: I > >> hope > >> > I can commit tomorrow. I could then > >> > > >> > 1. pick from your code what is missing in mine > >> > 2. apply it under your your commit --author > >> > 3. and finally update to copyright in the header comment to reflect > both > >> us > >> > > >> > Would that work for you? In step 1 there is a problem if I run into > code > >> > that stems back to the original contribution, which, if so, I can not > >> merge > >> > do to licensing. I hope that won't be the case. > >> > > >> > If you like, you can of course create a PR based on my commit > tomorrow, > >> you > >> > just need to make sure that all code that is added is originally > written > >> by > >> > you or under some license that is OK with ASL 2.0 (like the BSD > 2-clause > >> > from librdkafka). > >> > > >> > What would work for you? > >> > > >> > Rainer > >> > > >> > 2014-12-14 17:26 GMT+01:00 Rainer Gerhards >: > >> > > >> >> just sent it your way :-) > >> >> > >> >> 2014-12-14 17:24 GMT+01:00 Tait Clarridge : > >> >> > >> >>> > just leave it as is for now. I need to merge manually in any > case, as > >> >>> both > >> >>> > commits will be just a module. Bear a day with me, I don't want to > >> do a > >> >>> > hasty commit of my unfinished work. If you like, I can share a > >> working > >> >>> > copy, though. Just let me know. > >> >>> > > >> >>> > Rainer > >> >>> > >> >>> That sounds great, when you get a chance I'd like to take a look. > I'm > >> >>> in no rush as well so take your time :) > >> >>> > >> >>> Tait > >> >>> ___ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com/professional-services/ > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> >>> DON'T LIKE THAT. > >> >>> > >> >> > >> >> > >> > ___ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> ___ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUB
[rsyslog] ommongodb saves integer values as string
numeric values are not stored as integer values in the mongodb.
e.g. "bytes":"123" instead of "bytes":123
I tried it with the following template but had no success:
template(name="tpl1" type="list") {
constant(value="\"bytes\":")
property(name="$!bytes" type="int")
constant(value=", \"somestring\":\"")
property(name="$!somestring")
constant(value="\"")
}
action(type="ommongodb" server="127.0.0.1" db="test" collection="logs"
uid="test" pwd="" template="tpl1")
when I save it to a file (omfile) it makes it correctly (without the
quotes): "bytes":123
is this a known limitation of ommongodb that it stores all values as
strings? do I need to define the data type somewhere explicitly?
TIA
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] ommongodb saves integer values as string
numeric values are not stored as integer values in the mongodb.
e.g. "bytes":"123" instead of "bytes":123
I tried it with the following template but had no success:
template(name="tpl1" type="list") {
constant(value="\"bytes\":")
property(name="$!bytes" type="int")
constant(value=", \"somestring\":\"")
property(name="$!somestring")
constant(value="\"")
}
action(type="ommongodb" server="127.0.0.1" db="test" collection="logs"
uid="test" pwd="" template="tpl1")
when I save it to a file (omfile) it makes it correctly (without the
quotes): "bytes":123
is this a known limitation of ommongodb that it stores all values as
strings? do I need to define the data type somewhere explicitly?
TIA
--
View this message in context:
http://rsyslog-users.1305293.n2.nabble.com/ommongodb-saves-integer-values-as-string-tp7587211.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
