Re: [rsyslog] ruleset with only stop
2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
Today I noticed a ruleset with only stop as it's action will fail to
parse with rsyslog 8.4, but the same rule with a ~ will pass.
ruleset(name=testme) {
*.* ~
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
statement instead [try http://www.rsyslog.com/e/2307 ]
Changing to stop :
ruleset(name=testme) {
stop
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs will
run, but no output whatsoever is created. [try
http://www.rsyslog.com/e/2103
]
rsyslogd: run failed with error -2103 (see rsyslog.h or try
http://www.rsyslog.com/e/2103 to learn what that number means)
I have a situation where rules are being generated via templates in chef,
and having a rule that just discards messages would actually be a useful
thing to have, and ran into this.
So my question is, should a rule that only calls a discard action be
valid? If so, is this a bug in the config parser?
It's a little bit complex. The thing is that ~ actually *is* an action,
whereas stop is a statement. When I wrote that checking code, I never
envisioned that an empty ruleset could be useful for any case (if there is
just a stop inside it, it's practically empty, in that it simply does
nothing). I think in most cases this really is a config error. Maybe I
could add an permitEmpty parameter to the ruleset, which will then not
emit that error message.
To understand the whole picture: why do you need these empty rulesets?
Rainer
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] can someone lend me a hand on git procedure
2014-11-11 3:03 GMT+01:00 David Lang da...@lang.hm: On Mon, 10 Nov 2014, Thomas D. wrote: Hi, On 2014-11-08 00:19, David Lang wrote: Then when the feature is believed to be complete, it gets merged into a 'pending' tree. This tree exists for the sole purpose of having the testbench run against it daily. This tree is not stable, it's recreated for each day's test run by copying the master and then merging in all the pending features that are beleived to be ready to be tested. Mh... that's sounds like you don't always want to do QA. Then I disagree with you. QA should always be there. You shouldn't think about QA at all, because if we have CI, everything will happen automatically in the background. You don't need to care but if there is a QA problem, you will be notified. As a note on CI testing, no large entitiy does a full test run against each commit individually, this just doesn't scale. At some point the tests just take too long to run. So they fall back to doing the tests at a larer granularity, possibly with unit tests being run on each change (if you can define unit tests in a way that lets you decide easily what tests are relevant to the code that was changed) Unit tests should *always* run. Some functional tests maybe skipped and only run when you are preparing a release. But the discussion is very theoretical at the moment. Github's Travis is very powerful. Let's wait until we really know how much time rsyslog's test suite would actual take. it all depends on how long the tests take to run. If it takes a couple hours to run the tests, you aren't going to want to do them for every commit. Doing them frequently is important, but it's not so important that you need to throttle development to the speed that the tests can run. As for the history being dominated by merges, this is the result of maintaining so many different versions. As John noted when he did the documenation tree, merging changes into the oldest supported version and then merging that into the newer versions lets git do a LOT of work for you in how it remembers what changes have been made and integrating the new changes into it. Merging down instead of merging up would also help: I don't think so. Git already knows about the changes in the one direction, so layering some additional changes on isn't that hard, but moving the other way requires that git figure out where the change came from in order to 'do the right thing', and that's a much harder thing to do. ... especially as (to the best of my knowledge) git does not do that. So on the way down, you need to manually do the merging. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
2014-11-11 4:00 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Tue, Nov 11, 2014 at 7:43 AM, David Lang da...@lang.hm wrote: On Mon, 10 Nov 2014, Thomas D. wrote: Hi, On 2014-11-06 08:10, Rainer Gerhards wrote: Well, you can run the mysql test suite against any mysql server. It is independent from source. So that's not the best example at all... Don't you then need root (or similar) credentials to do some very delicate database operations like creating databases and tables? I wouldn't permit this to be done to my production server. That's why I asked for running the test suite in a 'sandbox'. you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Well, finally I installed valgrind and had to re-install glibc with debugging symbols (another error the test suite logged but did not detect). why did you need the debugging symbols? In Gentoo imdiag is currently only available when compiling with debug USE flag. This will also enable --enable-valgrind.. The error message logged in the udp-msgreduc-vg log file was something like valgrind: Fatal error at startup: a function redirection valgrind: which is mandatory for this platform-tool combination valgrind: cannot be set up. Details of the redirection are: valgrind: valgrind: [...] valgrind: valgrind: Possible fixes: (1, short term): install glibc's debuginfo valgrind: package on this machine. (2, longer term): ask the packagers valgrind: for your Linux distribution to please in future ship a non- valgrind: stripped ld.so (or whatever the dynamic linker .so is called) valgrind: that exports the above-named function using the standard valgrind: calling conventions for this platform. valgrind: valgrind: Cannot continue -- exiting now. Sorry. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features Given the fact that I am I and not a big team like at MySQL, I don't believe this is the right order of priorities. In reality, it is me alone doing 95% of all work, and funding for this work is almost solely by Adiscon (and only partially backed by paying customers). Asking this question shows me that you understand my point. Great! ;) Yes, I am voting for changing the priority order like you said. Don't get me wrong. I am not saying that you MUST change anything. I am only answering the question If you could change something in the project, what would you change? ... and I would change the priority to improve the current project quality using automatic tests. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) you could completely halt all documentation, bugfixing, and new features for a year while creating tests to cover everything. Testing is useful, but the end purpose of the software is not to pass tests, but rather to provide capabilities to the user. As an extreme example, I know of a company that lost a month of development and missed having the product ready for a national trade show because they got so caught up in writing tests for everything that a test that didn't work right when the software was being shutdown (after which it would be reset for the next test) HAD to be written before anything more could be done, and there was an upstream bug in one component that caused this exit to never work cleanly. Don't fall into a similar trap. +1 tests are just means to an end, we should be disciplined to write them as we go, but every project has corners(and sometimes large portions) that are ill-tested or worse. Paralysing development completely to test-cover them in one go doesn't sound like the best plan. FWIW, here is something that has worked for teams I have been on in the past. We were aware of which areas were not test-covered, bug-prone, hard-to-read etc. When we were to touch anything in those areas (like adding new features that touched those parts of code, or fix a bug etc), we used to test-cover directly and indirectly affected portions of that code, and then refactor them heavily to bring it up to the mark. I try to follow that same paradigm since quite a while. I have to admit it depends on workload, and some test are really *extremely* hard to create. For example, when you want to test out thing with the journal or imuxsock, you need to create a total mock environment and be
Re: [rsyslog] Feedback Request: do we still need -devel versions?
On Tue, 11 Nov 2014, Rainer Gerhards wrote: 2014-11-11 4:00 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Tue, Nov 11, 2014 at 7:43 AM, David Lang da...@lang.hm wrote: On Mon, 10 Nov 2014, Thomas D. wrote: Hi, On 2014-11-06 08:10, Rainer Gerhards wrote: Well, you can run the mysql test suite against any mysql server. It is independent from source. So that's not the best example at all... Don't you then need root (or similar) credentials to do some very delicate database operations like creating databases and tables? I wouldn't permit this to be done to my production server. That's why I asked for running the test suite in a 'sandbox'. you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Well, finally I installed valgrind and had to re-install glibc with debugging symbols (another error the test suite logged but did not detect). why did you need the debugging symbols? In Gentoo imdiag is currently only available when compiling with debug USE flag. This will also enable --enable-valgrind.. The error message logged in the udp-msgreduc-vg log file was something like valgrind: Fatal error at startup: a function redirection valgrind: which is mandatory for this platform-tool combination valgrind: cannot be set up. Details of the redirection are: valgrind: valgrind: [...] valgrind: valgrind: Possible fixes: (1, short term): install glibc's debuginfo valgrind: package on this machine. (2, longer term): ask the packagers valgrind: for your Linux distribution to please in future ship a non- valgrind: stripped ld.so (or whatever the dynamic linker .so is called) valgrind: that exports the above-named function using the standard valgrind: calling conventions for this platform. valgrind: valgrind: Cannot continue -- exiting now. Sorry. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features Given the fact that I am I and not a big team like at MySQL, I don't believe this is the right order of priorities. In reality, it is me alone doing 95% of all work, and funding for this work is almost solely by Adiscon (and only partially backed by paying customers). Asking this question shows me that you understand my point. Great! ;) Yes, I am voting for changing the priority order like you said. Don't get me wrong. I am not saying that you MUST change anything. I am only answering the question If you could change something in the project, what would you change? ... and I would change the priority to improve the current project quality using automatic tests. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) you could completely halt all documentation, bugfixing, and new features for a year while creating tests to cover everything. Testing is useful, but the end purpose of the software is not to pass tests, but rather to provide capabilities to the user. As an extreme example, I know of a company that lost a month of development and missed having the product ready for a national trade show because they got so caught up in writing tests for everything that a test that didn't work right when the software was being shutdown (after which it would be reset for the next test) HAD to be written before anything more could be done, and there was an upstream bug in one component that caused this exit to never work cleanly. Don't fall into a similar trap. +1 tests are just means to an end, we should be disciplined to write them as we go, but every project has corners(and sometimes large portions) that are ill-tested or worse. Paralysing development completely to test-cover them in one go doesn't sound like the best plan. FWIW, here is something that has worked for teams I have been on in the past. We were aware of which areas were not test-covered, bug-prone, hard-to-read etc. When we were to touch anything in those areas (like adding new features that touched those parts of code, or fix a bug etc), we used to test-cover directly and indirectly affected portions of that code, and then refactor them heavily to bring it up to the mark. I try to follow that same paradigm since quite a while. I have to admit it depends on workload, and some test are really *extremely* hard to create. For example, when you want to test out thing with the journal or imuxsock, you need to create a total mock environment and be extremely precise in when, how and where messages are injected and processed. Plus you need to make sure no other
[rsyslog] how to craft a test if the target behaves differently?
Hi experienced QA folks, I have one libdbi test that runs under valgrind (as do many other tests). The goal here is to automatically detect misadressings and memory leaks. The problem now is that different versions of libdbi have different memory leaks*. So on most current platforms, valgrind will report a memory. But that's a memory leak outside of my control. The usual approach is to use a suppression file. HOWEVER, the suppression file would need to be different for each platform/libdbi/dbserver combination. So this is not something I can ship as part of the testbench. Does anyone have an idea of how this could be solved? I will try now just to disable leak checking, so that the misaddressing protection is at least in place, but that's obviously a degradation of functionality. Help and feedback appreciated, Rainer * It's not a real leak in the case I am talking about, it has to do with when or when not libdbi calls mysql's thread shutdown functions. So this may even a problem that's solely related to the dbi driver layer. This is also nothing that needs to be reported to the libdbi project, because it definitely is fixed -- but as usual distros do not carry the fixed version. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
Hi, On 2014-11-11 03:13, David Lang wrote: you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Yes, running tests against any third-party application would require access to those applications while testing. Theses tests are called integration tests. Because you should *not* depend on those third-party applications you often cannot run your integration tests every time you run the normal test suite. Therefore you would split your tests. And this is already done for rsyslog: The mysql tests for example are separated. But this doesn't mean you cannot test the ommysql at all without mysql: You can write a mock so you can test your code at least. BTW: Travis will allow us to test against a running mysql each time... But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features [...] [...] Yes, I am voting for changing the priority order like you said. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) Wait, this order is not set in stone. For example, to write tests, you have to know what a component should do and how it should be used. So you need a documentation before you can write tests. you could completely halt all documentation, bugfixing, and new features for a year while creating tests to cover everything. Testing is useful, but the end purpose of the software is not to pass tests, but rather to provide capabilities to the user. As an extreme example, I know of a company that lost a month of development and missed having the product ready for a national trade show because they got so caught up in writing tests for everything that a test that didn't work right when the software was being shutdown (after which it would be reset for the next test) HAD to be written before anything more could be done, and there was an upstream bug in one component that caused this exit to never work cleanly. Don't fall into a similar trap. Our views are not so far apart. To be clear: I don't want to establish a process above the product. But the process should be part of the product. Change the way you currently work: When developing a new feature, start with the documentation. If you know how your new feature should look like, should be used, start writing tests so you can be sure your new feature is actual doing what you are are expecting. Now you can start the actual coding... In the end you have a new feature, fully documented and tested. If you don't follow this work flow, you will end up with the current situation: You maybe have great features -- but nobody knows due to missing/hard to use documentation. The tests will help you to keep your hard work working. Imagine you spend days for a new feature but another change will break it. Without tests you will only notice if somebody will actual try to use your feature and fill a bug. But we all know that this don't happen that often. Also, if you start looking for a new product which maybe solve your problem you are currently facing and experience a bug which is a show stopper for you in this moment, you will skip this product. So if Rainer should decide to switch the work flow like described, this would only help new features. But we still have to deal with the past. That's why I recommend to change priority for a moment. Like a sprint... like a bug hunt day/week/month where everybody focus on squashing bugs. If there is an upcoming national trade show where you want to show something new, yes... do that (but please follow the new work flow). But I am not aware of something like that in the next 4 months and I am not aware of a missing feature which must be implemented ASAP or rsyslog will go away... -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] how to craft a test if the target behaves differently?
Hi,
can't you write something like
# libdbi
{
Leak from dbi/dlopen
Memcheck:Leak
fun:calloc
obj:/lib/libdl-*.so
fun:dlopen
fun:dbi_initialize
}
which should match multiple libdbi versions?
-Thomas
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
On Tue, 11 Nov 2014, Thomas D. wrote: Hi, On 2014-11-11 03:13, David Lang wrote: you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Yes, running tests against any third-party application would require access to those applications while testing. Theses tests are called integration tests. Because you should *not* depend on those third-party applications you often cannot run your integration tests every time you run the normal test suite. Therefore you would split your tests. And this is already done for rsyslog: The mysql tests for example are separated. But this doesn't mean you cannot test the ommysql at all without mysql: You can write a mock so you can test your code at least. BTW: Travis will allow us to test against a running mysql each time... Add in Oracle, Postgres, RabbitMQ, 0MQ, journald, ElasticSearch, etc. The vast majority of the problems that we run into with rsyslog are in the interfaces to these third party destinations. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features [...] [...] Yes, I am voting for changing the priority order like you said. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) Wait, this order is not set in stone. For example, to write tests, you have to know what a component should do and how it should be used. So you need a documentation before you can write tests. snip Our views are not so far apart. To be clear: I don't want to establish a process above the product. But the process should be part of the product. Change the way you currently work: When developing a new feature, start with the documentation. If you know how your new feature should look like, should be used, start writing tests so you can be sure your new feature is actual doing what you are are expecting. Now you can start the actual coding... In the end you have a new feature, fully documented and tested. If you don't follow this work flow, you will end up with the current situation: You maybe have great features -- but nobody knows due to missing/hard to use documentation. The tests will help you to keep your hard work working. Imagine you spend days for a new feature but another change will break it. Without tests you will only notice if somebody will actual try to use your feature and fill a bug. But we all know that this don't happen that often. Also, if you start looking for a new product which maybe solve your problem you are currently facing and experience a bug which is a show stopper for you in this moment, you will skip this product. you _are_ argueing for process over product. You are pushing for strict test-driven-development, and while there are a lot of academics who really believe in such development, there are almost no development teams in the real world (commercial or opensource) that work that way for very long. The problem with documentation isn't that no documents get written (with the exception of some contributed modules), but rather that the documentation has been written by someone so close to the problem that there is a huge amount of implied context that isn't obvious to others, and that the organization of the documentation is based on how the person writing it thinks about the problems, not how the people who don't know about the software think about the problems. There are also cases that the author (both software and documentation) didn't think of when programming/wriring, and so those may not be documented. So if Rainer should decide to switch the work flow like described, this would only help new features. But we still have to deal with the past. That's why I recommend to change priority for a moment. Like a sprint... like a bug hunt day/week/month where everybody focus on squashing bugs. You keep missing that there isn't a lot of everybody in rsyslog development. Rainer is the author of 90% of both the code and the documentation. He is very willing to accept contributions, but we need to get other people involved. The split of the documentation into a different repository was to help ease the process of contributing documentation, but after a brief initial flurry, the outside contributions have tapered off. Please contribute tests, contribute documentation, contribute new features with any process that you want. And feel free to discuss development models, but recognize that there are a lot of them out there and that there is no one true way. Writing good tests is a lot
Re: [rsyslog] how to craft a test if the target behaves differently?
2014-11-11 14:26 GMT+01:00 Thomas D. whi...@whissi.de:
Hi,
can't you write something like
# libdbi
{
Leak from dbi/dlopen
Memcheck:Leak
fun:calloc
obj:/lib/libdl-*.so
fun:dlopen
fun:dbi_initialize
}
which should match multiple libdbi versions?
I gave it a try, doesn't seem to be so simple.
I let valgrind generate the suppressions to get started. This is for
Ubuntu14.04:
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:calloc
obj:*
obj:*
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
fun:actionPrepare
fun:processMsgMain
}
This is for CentOS 7:
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
obj:*
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
fun:actionPrepare
fun:processMsgMain
fun:doSubmitToActionQ
}
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
obj:*
obj:*
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
fun:actionPrepare
fun:processMsgMain
}{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:calloc
obj:*
obj:*
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
fun:actionPrepare
fun:processMsgMain
}
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
obj:*
obj:*
obj:*
fun:pthread_once
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
}
[I missed one violation here, but that doesn't really matter for the
context].
So it's quite different. I now thought I check if I misunderstood the
format and I can skip some of the contents. So on CentOS I changed
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
obj:*
obj:*
obj:*
fun:pthread_once
obj:*
obj:*
obj:*
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
}
to
{
insert_a_suppression_name_here
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
obj:*
obj:*
obj:*
fun:pthread_once
obj:*
fun:dbi_conn_connect
fun:initConn
fun:beginTransaction
}
... and the error shows up again.
Anything that I should change?
Rainer
-Thomas
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
2014-11-11 14:43 GMT+01:00 David Lang da...@lang.hm: On Tue, 11 Nov 2014, Thomas D. wrote: Hi, On 2014-11-11 03:13, David Lang wrote: you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Yes, running tests against any third-party application would require access to those applications while testing. Theses tests are called integration tests. Because you should *not* depend on those third-party applications you often cannot run your integration tests every time you run the normal test suite. Therefore you would split your tests. And this is already done for rsyslog: The mysql tests for example are separated. But this doesn't mean you cannot test the ommysql at all without mysql: You can write a mock so you can test your code at least. BTW: Travis will allow us to test against a running mysql each time... Add in Oracle, Postgres, RabbitMQ, 0MQ, journald, ElasticSearch, etc. The vast majority of the problems that we run into with rsyslog are in the interfaces to these third party destinations. I wouldn't go that far to say it's the vast majority, but its a big bunch in any case. A common case, however, is that the most useful tests are those that are very hard to create - most notably those things that deal with races. I judge most useful by actual bug reports which could have been avoided by a test. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features [...] [...] Yes, I am voting for changing the priority order like you said. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) Wait, this order is not set in stone. For example, to write tests, you have to know what a component should do and how it should be used. So you need a documentation before you can write tests. snip Our views are not so far apart. To be clear: I don't want to establish a process above the product. But the process should be part of the product. Change the way you currently work: When developing a new feature, start with the documentation. If you know how your new feature should look like, should be used, start writing tests so you can be sure your new feature is actual doing what you are are expecting. Now you can start the actual coding... In the end you have a new feature, fully documented and tested. If you don't follow this work flow, you will end up with the current situation: You maybe have great features -- but nobody knows due to missing/hard to use documentation. The tests will help you to keep your hard work working. Imagine you spend days for a new feature but another change will break it. Without tests you will only notice if somebody will actual try to use your feature and fill a bug. But we all know that this don't happen that often. Also, if you start looking for a new product which maybe solve your problem you are currently facing and experience a bug which is a show stopper for you in this moment, you will skip this product. you _are_ argueing for process over product. You are pushing for strict test-driven-development, and while there are a lot of academics who really believe in such development, there are almost no development teams in the real world (commercial or opensource) that work that way for very long. The problem with documentation isn't that no documents get written (with the exception of some contributed modules), but rather that the documentation has been written by someone so close to the problem that there is a huge amount of implied context that isn't obvious to others, and that the organization of the documentation is based on how the person writing it thinks about the problems, not how the people who don't know about the software think about the problems. There are also cases that the author (both software and documentation) didn't think of when programming/wriring, and so those may not be documented. +1 So if Rainer should decide to switch the work flow like described, this would only help new features. But we still have to deal with the past. That's why I recommend to change priority for a moment. Like a sprint... like a bug hunt day/week/month where everybody focus on squashing bugs. You keep missing that there isn't a lot of everybody in rsyslog development. Rainer is the author of 90% of both the code and the documentation. He is very willing to accept contributions, but we need to get other people involved. The split of the documentation
Re: [rsyslog] Feedback Request: do we still need -devel versions?
side-note: It looks that with my last commit, the testbench in master branch should now work. I couldn't test all possible option/platform combinations. So it would be good if someone (Thomas? Anyone else?) could try to break it and report back. Thanks, Rainer 2014-11-11 15:20 GMT+01:00 Rainer Gerhards rgerha...@hq.adiscon.com: 2014-11-11 14:43 GMT+01:00 David Lang da...@lang.hm: On Tue, 11 Nov 2014, Thomas D. wrote: Hi, On 2014-11-11 03:13, David Lang wrote: you still need those credentials for your sandbox, right? If MySQL isn't installed on the system you will need to install the package (requiring root + distro specific knowledge), if it is installed, you need to configure users and permissions for the test user (requiring DB root equivalent + distro specific knowlege) Yes, running tests against any third-party application would require access to those applications while testing. Theses tests are called integration tests. Because you should *not* depend on those third-party applications you often cannot run your integration tests every time you run the normal test suite. Therefore you would split your tests. And this is already done for rsyslog: The mysql tests for example are separated. But this doesn't mean you cannot test the ommysql at all without mysql: You can write a mock so you can test your code at least. BTW: Travis will allow us to test against a running mysql each time... Add in Oracle, Postgres, RabbitMQ, 0MQ, journald, ElasticSearch, etc. The vast majority of the problems that we run into with rsyslog are in the interfaces to these third party destinations. I wouldn't go that far to say it's the vast majority, but its a big bunch in any case. A common case, however, is that the most useful tests are those that are very hard to create - most notably those things that deal with races. I judge most useful by actual bug reports which could have been avoided by a test. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features [...] [...] Yes, I am voting for changing the priority order like you said. I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 lives is subject to a lot of debate) Wait, this order is not set in stone. For example, to write tests, you have to know what a component should do and how it should be used. So you need a documentation before you can write tests. snip Our views are not so far apart. To be clear: I don't want to establish a process above the product. But the process should be part of the product. Change the way you currently work: When developing a new feature, start with the documentation. If you know how your new feature should look like, should be used, start writing tests so you can be sure your new feature is actual doing what you are are expecting. Now you can start the actual coding... In the end you have a new feature, fully documented and tested. If you don't follow this work flow, you will end up with the current situation: You maybe have great features -- but nobody knows due to missing/hard to use documentation. The tests will help you to keep your hard work working. Imagine you spend days for a new feature but another change will break it. Without tests you will only notice if somebody will actual try to use your feature and fill a bug. But we all know that this don't happen that often. Also, if you start looking for a new product which maybe solve your problem you are currently facing and experience a bug which is a show stopper for you in this moment, you will skip this product. you _are_ argueing for process over product. You are pushing for strict test-driven-development, and while there are a lot of academics who really believe in such development, there are almost no development teams in the real world (commercial or opensource) that work that way for very long. The problem with documentation isn't that no documents get written (with the exception of some contributed modules), but rather that the documentation has been written by someone so close to the problem that there is a huge amount of implied context that isn't obvious to others, and that the organization of the documentation is based on how the person writing it thinks about the problems, not how the people who don't know about the software think about the problems. There are also cases that the author (both software and documentation) didn't think of when programming/wriring, and so those may not be documented. +1 So if Rainer should decide to switch the work flow like described, this would only help new features. But we still have to deal with the past. That's why I recommend to change priority for a moment. Like a sprint...
Re: [rsyslog] ruleset with only stop
Rainer,
I agree that an empty ruleset is an oddity. In our case, the short answer
is that we are generating configurations from templates using chef, and the
ability to have a ruleset that simply discards would make part of that
process much simpler for us.
It is admittedly an edge case.
Brian
On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:
2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
Today I noticed a ruleset with only stop as it's action will fail to
parse with rsyslog 8.4, but the same rule with a ~ will pass.
ruleset(name=testme) {
*.* ~
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
statement instead [try http://www.rsyslog.com/e/2307 ]
Changing to stop :
ruleset(name=testme) {
stop
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
will
run, but no output whatsoever is created. [try
http://www.rsyslog.com/e/2103
]
rsyslogd: run failed with error -2103 (see rsyslog.h or try
http://www.rsyslog.com/e/2103 to learn what that number means)
I have a situation where rules are being generated via templates in chef,
and having a rule that just discards messages would actually be a useful
thing to have, and ran into this.
So my question is, should a rule that only calls a discard action be
valid? If so, is this a bug in the config parser?
It's a little bit complex. The thing is that ~ actually *is* an action,
whereas stop is a statement. When I wrote that checking code, I never
envisioned that an empty ruleset could be useful for any case (if there is
just a stop inside it, it's practically empty, in that it simply does
nothing). I think in most cases this really is a config error. Maybe I
could add an permitEmpty parameter to the ruleset, which will then not
emit that error message.
To understand the whole picture: why do you need these empty rulesets?
Rainer
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] ruleset with only stop
On Tue, 11 Nov 2014, Brian Knox wrote:
Rainer,
I agree that an empty ruleset is an oddity. In our case, the short answer
is that we are generating configurations from templates using chef, and the
ability to have a ruleset that simply discards would make part of that
process much simpler for us.
It is admittedly an edge case.
It's an edge case, but I think it's one that should be supported if possible.
automated config generation is very useful, and being able to group rules into
rulesets and call them with the calling function not having any idea of what is
going to happen with the logs at that point is very useful, being able to have a
high level config split the logs up and call different rulesets on different
logs without having to worry if the ruleset does something with the logs or just
throws them away is _very_ useful.
So it is a corner case, but I think it's a valuable one to support.
I would suggest that at config load time, that this is an odd enough corner case
that it's worth logging a ruleset X can't do anything with the logs message,
not just for the case where the only action is to throw it away, but also for
the cases where the conditions in a ruleset cannot possibly match any log
message (if priority = info then *.crit also cannot match anything for example)
David Lang
Brian
On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:
2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
Today I noticed a ruleset with only stop as it's action will fail to
parse with rsyslog 8.4, but the same rule with a ~ will pass.
ruleset(name=testme) {
*.* ~
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
statement instead [try http://www.rsyslog.com/e/2307 ]
Changing to stop :
ruleset(name=testme) {
stop
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
will
run, but no output whatsoever is created. [try
http://www.rsyslog.com/e/2103
]
rsyslogd: run failed with error -2103 (see rsyslog.h or try
http://www.rsyslog.com/e/2103 to learn what that number means)
I have a situation where rules are being generated via templates in chef,
and having a rule that just discards messages would actually be a useful
thing to have, and ran into this.
So my question is, should a rule that only calls a discard action be
valid? If so, is this a bug in the config parser?
It's a little bit complex. The thing is that ~ actually *is* an action,
whereas stop is a statement. When I wrote that checking code, I never
envisioned that an empty ruleset could be useful for any case (if there is
just a stop inside it, it's practically empty, in that it simply does
nothing). I think in most cases this really is a config error. Maybe I
could add an permitEmpty parameter to the ruleset, which will then not
emit that error message.
To understand the whole picture: why do you need these empty rulesets?
Rainer
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] ruleset with only stop
2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
On Tue, 11 Nov 2014, Brian Knox wrote:
Rainer,
I agree that an empty ruleset is an oddity. In our case, the short answer
is that we are generating configurations from templates using chef, and
the
ability to have a ruleset that simply discards would make part of that
process much simpler for us.
It is admittedly an edge case.
It's an edge case, but I think it's one that should be supported if
possible.
automated config generation is very useful, and being able to group rules
into rulesets and call them with the calling function not having any idea
of what is going to happen with the logs at that point is very useful,
being able to have a high level config split the logs up and call different
rulesets on different logs without having to worry if the ruleset does
something with the logs or just throws them away is _very_ useful.
So it is a corner case, but I think it's a valuable one to support.
ack
I would suggest that at config load time, that this is an odd enough
corner case that it's worth logging a ruleset X can't do anything with the
logs message, not just for the case where the only action is to throw it
away, but also for the cases where the conditions in a ruleset cannot
possibly match any log message (if priority = info then *.crit also cannot
match anything for example)
Let's start with what we have on the table. I think it is best to add a
ruleset parameter permitEmpty=on. That way, we don't generate
error/warning messages when the user is aware of what he is doing. In any
manual case (without config gen), I'd still say that's an error indication.
On the topic of no filter matches. That's quite complex, as you would need
to evaluate all the filters and possible conditions. Not sure if it can
even reliably done. Am I overlooking something?
Rainer
David Lang
Brian
On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards
rgerha...@hq.adiscon.com
wrote:
2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
Today I noticed a ruleset with only stop as it's action will fail to
parse with rsyslog 8.4, but the same rule with a ~ will pass.
ruleset(name=testme) {
*.* ~
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
statement instead [try http://www.rsyslog.com/e/2307 ]
Changing to stop :
ruleset(name=testme) {
stop
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
will
run, but no output whatsoever is created. [try
http://www.rsyslog.com/e/2103
]
rsyslogd: run failed with error -2103 (see rsyslog.h or try
http://www.rsyslog.com/e/2103 to learn what that number means)
I have a situation where rules are being generated via templates in
chef,
and having a rule that just discards messages would actually be a useful
thing to have, and ran into this.
So my question is, should a rule that only calls a discard action be
valid? If so, is this a bug in the config parser?
It's a little bit complex. The thing is that ~ actually *is* an
action,
whereas stop is a statement. When I wrote that checking code, I never
envisioned that an empty ruleset could be useful for any case (if there
is
just a stop inside it, it's practically empty, in that it simply does
nothing). I think in most cases this really is a config error. Maybe I
could add an permitEmpty parameter to the ruleset, which will then not
emit that error message.
To understand the whole picture: why do you need these empty rulesets?
Rainer
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
Re: [rsyslog] ruleset with only stop
+1 for permitEmpty=on - it would definitely simplify our lives.
Brian
On Tue, Nov 11, 2014 at 11:40 AM, Boylan, James james.boy...@orbitz.com
wrote:
I think that the permitEmpty=on field is a reasonable starting place. I
have a config management app that I use with rsyslog that this field would
help significantly with.
-- James
--- Sent from my mobile phone ---
- Reply message -
From: Rainer Gerhards rgerha...@hq.adiscon.com
To: rsyslog-users rsyslog@lists.adiscon.com
Subject: [rsyslog] ruleset with only stop
Date: Tue, Nov 11, 2014 10:29 AM
2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
On Tue, 11 Nov 2014, Brian Knox wrote:
Rainer,
I agree that an empty ruleset is an oddity. In our case, the short
answer
is that we are generating configurations from templates using chef, and
the
ability to have a ruleset that simply discards would make part of that
process much simpler for us.
It is admittedly an edge case.
It's an edge case, but I think it's one that should be supported if
possible.
automated config generation is very useful, and being able to group rules
into rulesets and call them with the calling function not having any idea
of what is going to happen with the logs at that point is very useful,
being able to have a high level config split the logs up and call
different
rulesets on different logs without having to worry if the ruleset does
something with the logs or just throws them away is _very_ useful.
So it is a corner case, but I think it's a valuable one to support.
ack
I would suggest that at config load time, that this is an odd enough
corner case that it's worth logging a ruleset X can't do anything with
the
logs message, not just for the case where the only action is to throw it
away, but also for the cases where the conditions in a ruleset cannot
possibly match any log message (if priority = info then *.crit also
cannot
match anything for example)
Let's start with what we have on the table. I think it is best to add a
ruleset parameter permitEmpty=on. That way, we don't generate
error/warning messages when the user is aware of what he is doing. In any
manual case (without config gen), I'd still say that's an error indication.
On the topic of no filter matches. That's quite complex, as you would need
to evaluate all the filters and possible conditions. Not sure if it can
even reliably done. Am I overlooking something?
Rainer
David Lang
Brian
On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards
rgerha...@hq.adiscon.com
wrote:
2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
Today I noticed a ruleset with only stop as it's action will fail to
parse with rsyslog 8.4, but the same rule with a ~ will pass.
ruleset(name=testme) {
*.* ~
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master
config
./test.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
statement instead [try http://www.rsyslog.com/e/2307 ]
Changing to stop :
ruleset(name=testme) {
stop
}
bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
rsyslogd: version 8.5.0, config validation run (level 1), master
config
./test.conf
rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
will
run, but no output whatsoever is created. [try
http://www.rsyslog.com/e/2103
]
rsyslogd: run failed with error -2103 (see rsyslog.h or try
http://www.rsyslog.com/e/2103 to learn what that number means)
I have a situation where rules are being generated via templates in
chef,
and having a rule that just discards messages would actually be a
useful
thing to have, and ran into this.
So my question is, should a rule that only calls a discard action be
valid? If so, is this a bug in the config parser?
It's a little bit complex. The thing is that ~ actually *is* an
action,
whereas stop is a statement. When I wrote that checking code, I never
envisioned that an empty ruleset could be useful for any case (if there
is
just a stop inside it, it's practically empty, in that it simply does
nothing). I think in most cases this really is a config error. Maybe I
could add an permitEmpty parameter to the ruleset, which will then
not
emit that error message.
To understand the whole picture: why do you need these empty rulesets?
Rainer
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog
Re: [rsyslog] Feedback Request: do we still need -devel versions?
On 2014-11-11 22:02, Thomas D. wrote: Not sure about libgcc_s.so.1 must be installed for pthread_cancel to work = # locate libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/32/libgcc_s.so.1 and # pwd /var/tmp/portage/app-admin/rsyslog-8./work/rsyslog-8./tests # srcdir=. ./imptcp_conndrop.sh TEST: [imptcp_conndrop.sh]: test imptcp with random connection drops rsyslogd started with pid 3462 00020 open connections starting run 1 Sending 5 messages. 0005 messages sent runtime: 2.442 00020 close connections -D option initiated 2477 connection closures End of tcpflood Run imdiag[13500]: mainqueue empty libgcc_s.so.1 must be installed for pthread_cancel to work ./diag.sh: line 16: 3462 Aborted (core dumped) $valgrind ../tools/rsyslogd -u2 -n -irsyslog$3.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 Back trace: #0 0x7f623d0fcec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x7f623d0fe2ca in __GI_abort () at abort.c:89 #2 0x0042fd8f in sigsegvHdlr (signum=6) at debug.c:852 #3 signal handler called #4 0x7f623d0fcec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #5 0x7f623d0fe2ca in __GI_abort () at abort.c:89 #6 0x7f623d13af34 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f623d22b3cf %s) at ../sysdeps/posix/libc_fatal.c:175 #7 0x7f623d13af5e in __GI___libc_fatal (message=0x7f623e2cb3d8 libgcc_s.so.1 must be installed for pthread_cancel to work\n) at ../sysdeps/posix/libc_fatal.c:186 #8 0x7f623e2c9864 in pthread_cancel_init () at ../sysdeps/nptl/unwind-forcedunwind.c:64 #9 0x7f623e2c994c in _Unwind_ForcedUnwind (exc=0x7f623bdd0d70, stop=stop@entry=0x7f623e2c7c60 unwind_stop, stop_argument=0x7f623bdcfeb0) at ../sysdeps/nptl/unwind-forcedunwind.c:129 #10 0x7f623e2c7dd0 in __GI___pthread_unwind (buf=optimized out) at unwind.c:129 #11 0x7f623e2c2465 in __do_cancel () at pthreadP.h:277 #12 __pthread_exit (value=optimized out) at pthread_exit.c:29 #13 0x0045df82 in thrdStarter (arg=0x7f623c82d3e0) at ../threads.c:227 #14 0x7f623e2c13c4 in start_thread (arg=0x7f623bdd0700) at pthread_create.c:310 #15 0x7f623d1ada2d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Using glibc-2.20. Related to: https://sourceware.org/bugzilla/show_bug.cgi?id=13119 ? At least when running the example from the bug report I am getting a similar back trace: #0 0x7fc39c017ec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x7fc39c0192ca in __GI_abort () at abort.c:89 #2 0x7fc39c055f34 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7fc39c1463cf %s) at ../sysdeps/posix/libc_fatal.c:175 #3 0x7fc39c055f5e in __GI___libc_fatal (message=0x7fc39c3903d8 libgcc_s.so.1 must be installed for pthread_cancel to work\n) at ../sysdeps/posix/libc_fatal.c:186 #4 0x7fc39c38e864 in pthread_cancel_init () at ../sysdeps/nptl/unwind-forcedunwind.c:64 #5 0x7fc39c38e94c in _Unwind_ForcedUnwind (exc=0x7fc39bfe2d70, stop=stop@entry=0x7fc39c38cc60 unwind_stop, stop_argument=0x7fc39bfe1f70) at ../sysdeps/nptl/unwind-forcedunwind.c:129 #6 0x7fc39c38cdd0 in __GI___pthread_unwind (buf=optimized out) at unwind.c:129 #7 0x7fc39c387465 in __do_cancel () at pthreadP.h:277 #8 __pthread_exit (value=optimized out) at pthread_exit.c:29 #9 0x00400721 in foo (ptr=0x0) at foo.c:15 #10 0x7fc39c3863c4 in start_thread (arg=0x7fc39bfe2700) at pthread_create.c:310 #11 0x7fc39c0c8a2d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
If thread _ cancel does not work all sorts of things can go wrong. Do you know why the platform is defective? Any way to detect this? So that we during configure can tell it does not work? Rainer Sent from phone, thus brief. Am 11.11.2014 22:21 schrieb Thomas D. whi...@whissi.de: On 2014-11-11 22:02, Thomas D. wrote: Not sure about libgcc_s.so.1 must be installed for pthread_cancel to work = # locate libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/32/libgcc_s.so.1 and # pwd /var/tmp/portage/app-admin/rsyslog-8./work/rsyslog-8./tests # srcdir=. ./imptcp_conndrop.sh TEST: [imptcp_conndrop.sh]: test imptcp with random connection drops rsyslogd started with pid 3462 00020 open connections starting run 1 Sending 5 messages. 0005 messages sent runtime: 2.442 00020 close connections -D option initiated 2477 connection closures End of tcpflood Run imdiag[13500]: mainqueue empty libgcc_s.so.1 must be installed for pthread_cancel to work ./diag.sh: line 16: 3462 Aborted (core dumped) $valgrind ../tools/rsyslogd -u2 -n -irsyslog$3.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 Back trace: #0 0x7f623d0fcec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x7f623d0fe2ca in __GI_abort () at abort.c:89 #2 0x0042fd8f in sigsegvHdlr (signum=6) at debug.c:852 #3 signal handler called #4 0x7f623d0fcec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #5 0x7f623d0fe2ca in __GI_abort () at abort.c:89 #6 0x7f623d13af34 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f623d22b3cf %s) at ../sysdeps/posix/libc_fatal.c:175 #7 0x7f623d13af5e in __GI___libc_fatal (message=0x7f623e2cb3d8 libgcc_s.so.1 must be installed for pthread_cancel to work\n) at ../sysdeps/posix/libc_fatal.c:186 #8 0x7f623e2c9864 in pthread_cancel_init () at ../sysdeps/nptl/unwind-forcedunwind.c:64 #9 0x7f623e2c994c in _Unwind_ForcedUnwind (exc=0x7f623bdd0d70, stop=stop@entry=0x7f623e2c7c60 unwind_stop, stop_argument=0x7f623bdcfeb0) at ../sysdeps/nptl/unwind-forcedunwind.c:129 #10 0x7f623e2c7dd0 in __GI___pthread_unwind (buf=optimized out) at unwind.c:129 #11 0x7f623e2c2465 in __do_cancel () at pthreadP.h:277 #12 __pthread_exit (value=optimized out) at pthread_exit.c:29 #13 0x0045df82 in thrdStarter (arg=0x7f623c82d3e0) at ../threads.c:227 #14 0x7f623e2c13c4 in start_thread (arg=0x7f623bdd0700) at pthread_create.c:310 #15 0x7f623d1ada2d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Using glibc-2.20. Related to: https://sourceware.org/bugzilla/show_bug.cgi?id=13119 ? At least when running the example from the bug report I am getting a similar back trace: #0 0x7fc39c017ec7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x7fc39c0192ca in __GI_abort () at abort.c:89 #2 0x7fc39c055f34 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7fc39c1463cf %s) at ../sysdeps/posix/libc_fatal.c:175 #3 0x7fc39c055f5e in __GI___libc_fatal (message=0x7fc39c3903d8 libgcc_s.so.1 must be installed for pthread_cancel to work\n) at ../sysdeps/posix/libc_fatal.c:186 #4 0x7fc39c38e864 in pthread_cancel_init () at ../sysdeps/nptl/unwind-forcedunwind.c:64 #5 0x7fc39c38e94c in _Unwind_ForcedUnwind (exc=0x7fc39bfe2d70, stop=stop@entry=0x7fc39c38cc60 unwind_stop, stop_argument=0x7fc39bfe1f70) at ../sysdeps/nptl/unwind-forcedunwind.c:129 #6 0x7fc39c38cdd0 in __GI___pthread_unwind (buf=optimized out) at unwind.c:129 #7 0x7fc39c387465 in __do_cancel () at pthreadP.h:277 #8 __pthread_exit (value=optimized out) at pthread_exit.c:29 #9 0x00400721 in foo (ptr=0x0) at foo.c:15 #10 0x7fc39c3863c4 in start_thread (arg=0x7fc39bfe2700) at pthread_create.c:310 #11 0x7fc39c0c8a2d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
Re: [rsyslog] ruleset with only stop
On Tue, 11 Nov 2014, Rainer Gerhards wrote: 2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm: On Tue, 11 Nov 2014, Brian Knox wrote: Rainer, I agree that an empty ruleset is an oddity. In our case, the short answer is that we are generating configurations from templates using chef, and the ability to have a ruleset that simply discards would make part of that process much simpler for us. It is admittedly an edge case. It's an edge case, but I think it's one that should be supported if possible. automated config generation is very useful, and being able to group rules into rulesets and call them with the calling function not having any idea of what is going to happen with the logs at that point is very useful, being able to have a high level config split the logs up and call different rulesets on different logs without having to worry if the ruleset does something with the logs or just throws them away is _very_ useful. So it is a corner case, but I think it's a valuable one to support. ack I would suggest that at config load time, that this is an odd enough corner case that it's worth logging a ruleset X can't do anything with the logs message, not just for the case where the only action is to throw it away, but also for the cases where the conditions in a ruleset cannot possibly match any log message (if priority = info then *.crit also cannot match anything for example) Let's start with what we have on the table. I think it is best to add a ruleset parameter permitEmpty=on. That way, we don't generate error/warning messages when the user is aware of what he is doing. In any manual case (without config gen), I'd still say that's an error indication. I think that this is a sufficently corner case that I'm not sure it's worth the extra complexity to silence the warning. I think that people who do this intentionally can just ignore the log message. On the topic of no filter matches. That's quite complex, as you would need to evaluate all the filters and possible conditions. Not sure if it can even reliably done. Am I overlooking something? I am not saying that it should try to catch every possible case, but I was thinking that the configuration optimization step would optomize away some impossible combinations, and that could result in an empty ruleset. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
if you can find the diag.sh that it's using, look at what binaries it's calling and do an ldd on them, it's likely that that library isn't in the ldconfig path. David Lang On Tue, 11 Nov 2014, Thomas D. wrote: Date: Tue, 11 Nov 2014 22:02:41 +0100 From: Thomas D. whi...@whissi.de Reply-To: rsyslog-users rsyslog@lists.adiscon.com To: rsyslog@lists.adiscon.com Subject: Re: [rsyslog] Feedback Request: do we still need -devel versions? Hi, On 2014-11-11 15:24, Rainer Gerhards wrote: side-note: It looks that with my last commit, the testbench in master branch should now work. I couldn't test all possible option/platform combinations. So it would be good if someone (Thomas? Anyone else?) could try to break it and report back. imptcp_conndrop.sh is failing, but this time the test suite won't wait forever, nice! ;) [...] PASS: failover-no-rptd-vg.sh PASS: udp-msgreduc-vg.sh PASS: udp-msgreduc-orgmsg-vg.sh PASS: tcp-msgreduc-vg.sh PASS: manyptcp.sh PASS: imptcp_large.sh PASS: imptcp_addtlframedelim.sh libgcc_s.so.1 must be installed for pthread_cancel to work FAIL: imptcp_conndrop.sh PASS: rscript_replace.sh PASS: rscript_replace_complex.sh PASS: rscript_wrap2.sh PASS: rscript_wrap3.sh [...] Testsuite summary for rsyslog 8.5.0 # TOTAL: 124 # PASS: 118 # SKIP: 5 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 See tests/test-suite.log Please report to rsyslog@lists.adiscon.com tests/test-suite.log: https://bpaste.net/show/a488b224cfd9 rsyslog was build with: /var/tmp/portage/app-admin/rsyslog-8./work/rsyslog-8./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --disable-dependency-tracking --disable-silent-rules --libdir=/usr/lib64 --docdir=/usr/share/doc/rsyslog-8. --enable-gene rate-man-pages --enable-imfile --enable-impstats --enable-imptcp --enable-imttcp --enable-mmanon --enable-mmaudit --enable-mmfields --enable-mmjsonparse --enable-mmpstrucdata --enable-mmsequ ence --enable-mmutf8fix --enable-mail --enable-omprog --enable-omruleset --enable-omstdout --enable-omuxsock --enable-pmaixforwardedfrom --enable-pmciscoios --enable-pmcisconames --enable-pm lastmsg --enable-pmrfc3164sd --enable-pmsnare --disable-libdbi --disable-ommongodb --disable-mysql --disable-oracle --disable-pgsql --disable-omhiredis --enable-debug --enable-diagtools --en able-imdiag --enable-memcheck --enable-rtinst --enable-valgrind --disable-elasticsearch --enable-libgcrypt --enable-jemalloc --disable-gssapi-krb5 --disable-mmnormalize --disable-omudpspoof --disable-omrabbitmq --disable-relp --disable-rfc3195 --disable-mmrfc5424addhmac --disable-snmp --disable-mmsnmptrapd --disable-gnutls --disable-imjournal --disable-omjournal --disable-usert ools --disable-imzmq3 --disable-omzmq3 --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-testbench Not sure about libgcc_s.so.1 must be installed for pthread_cancel to work = # locate libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.3/32/libgcc_s.so.1 and # pwd /var/tmp/portage/app-admin/rsyslog-8./work/rsyslog-8./tests # srcdir=. ./imptcp_conndrop.sh TEST: [imptcp_conndrop.sh]: test imptcp with random connection drops rsyslogd started with pid 3462 00020 open connections starting run 1 Sending 5 messages. 0005 messages sent runtime: 2.442 00020 close connections -D option initiated 2477 connection closures End of tcpflood Run imdiag[13500]: mainqueue empty libgcc_s.so.1 must be installed for pthread_cancel to work ./diag.sh: line 16: 3462 Aborted (core dumped) $valgrind ../tools/rsyslogd -u2 -n -irsyslog$3.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
Re: [rsyslog] ruleset with only stop
If was able to use an empty ruleset, a warning resulting from that wouldn't bother me at all. Brian On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm wrote: On Tue, 11 Nov 2014, Rainer Gerhards wrote: 2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm: On Tue, 11 Nov 2014, Brian Knox wrote: Rainer, I agree that an empty ruleset is an oddity. In our case, the short answer is that we are generating configurations from templates using chef, and the ability to have a ruleset that simply discards would make part of that process much simpler for us. It is admittedly an edge case. It's an edge case, but I think it's one that should be supported if possible. automated config generation is very useful, and being able to group rules into rulesets and call them with the calling function not having any idea of what is going to happen with the logs at that point is very useful, being able to have a high level config split the logs up and call different rulesets on different logs without having to worry if the ruleset does something with the logs or just throws them away is _very_ useful. So it is a corner case, but I think it's a valuable one to support. ack I would suggest that at config load time, that this is an odd enough corner case that it's worth logging a ruleset X can't do anything with the logs message, not just for the case where the only action is to throw it away, but also for the cases where the conditions in a ruleset cannot possibly match any log message (if priority = info then *.crit also cannot match anything for example) Let's start with what we have on the table. I think it is best to add a ruleset parameter permitEmpty=on. That way, we don't generate error/warning messages when the user is aware of what he is doing. In any manual case (without config gen), I'd still say that's an error indication. I think that this is a sufficently corner case that I'm not sure it's worth the extra complexity to silence the warning. I think that people who do this intentionally can just ignore the log message. On the topic of no filter matches. That's quite complex, as you would need to evaluate all the filters and possible conditions. Not sure if it can even reliably done. Am I overlooking something? I am not saying that it should try to catch every possible case, but I was thinking that the configuration optimization step would optomize away some impossible combinations, and that could result in an empty ruleset. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
On 2014-11-11 22:27, David Lang wrote: if you can find the diag.sh that it's using, look at what binaries it's calling and do an ldd on them, it's likely that that library isn't in the ldconfig path. To make sure the build environment doesn't have the problem (because when I manually run the test, the test seems to pass) I added some debugging code to imptcp_conndrop.sh, see https://bpaste.net/show/c52e4da0a446 Now the output looks like https://bpaste.net/show/b35599ee85c0 Looks good, not? ...and the manual run passed again: http://pastebin.com/raw.php?i=niXvTeUt Not sure what's going on :/ -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Feedback Request: do we still need -devel versions?
On 2014-11-11 22:24, Rainer Gerhards wrote: If thread _ cancel does not work all sorts of things can go wrong. Do you know why the platform is defective? Any way to detect this? So that we during configure can tell it does not work? At the moment I don't understand what's going on. The core dump created in rsyslog's test seems to be identical with the one you'll get when running the code from glibc's bug tracker like I quoted in my previous mail. But because when I manually run the failed rsyslog test again after make check the test will pass... no pthread_cancel error. So *I* cannot tell you at the moment if there's a problem with pthread_cancel at all you need to get around. -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
