On Fri, Nov 3, 2017 at 12:21 AM, Cédric Jeanneret <cedric.jeanne...@camptocamp.com> wrote: > On 11/02/2017 05:18 PM, Ben Nemec wrote: >> Adding tags for the relevant projects. I _think_ this is mostly >> directed at Puppet/TripleO, but Oslo is obviously relevant as well. > > Thank you! First mail in there, wasn't really sure how to do that. > >> >> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote: >>> Dear Stackers, >>> >>> While working on my locally deployed Openstack (Pike using TripleO), I >>> was a bit struggling with the logging part. Currently, all logs are >>> pushed to per-service files, in the standard format "one line per >>> entry", plain flat text. >>> >>> It's nice, but if one is wanting to push and index those logs in an ELK, >>> the current, default format isn't really good. >>> >>> After some discussions about oslo.log, it appears it provides a nice >>> JSONFormatter handler¹ one might want to use for each (python) service >>> using oslo central library. >>> >>> A JSON format is really cool, as it's easy to parse for machines, and it >>> can be on a multi-line without any bit issue - this is especially >>> important for stack traces, as their output is multi-line without real >>> way to have a common delimiter so that we can re-format it and feed it >>> to any log parser (logstash, fluentd, …). >>> >>> After some more talks, olso.log will not provide a unified interface in >>> order to output all received logs as JSON - this makes sens, as that >>> would mean "rewrite almost all the python logging management >>> interface"², and that's pretty useless, since (all?) services have their >>> own "logging.conf" file. >>> >>> That said… to the main purpose of that mail: >>> >>> - Default format for logs >>> A first question would be "are we all OK with the default output format" >>> - I'm pretty sure "humans" are happy with that, as it's really >>> convenient to read and grep. But on a "standard" Openstack deploy, I'm >>> pretty sure one does not have only one controller, one ceph node and one >>> compute. Hence comes the log centralization, and with that, the log >>> indexation and treatments. >>> >>> For that, one might argue "I'm using plain files on my logger, and >>> grep-ing -r in them". That's a way to do things, and for that, plain, >>> flat logs are great. >>> >>> But… I'm pretty sure I'm not the only one wanting to use some kind of >>> ELK cluster for that kind of purpose. So the right question is: what >>> about switching the default log format to JSON? On my part, I don't see >>> "cons", only "pros", but my judgment is of course biased, as I'm "alone >>> in my corner". But what about you, Community? >> >> The major con I see is that we don't require an ELK stack and reading >> JSON logs if you don't have one of those is probably worse, although I >> will admit I haven't spent much time reading our JSON-formatted logs. My >> experience with things that log in JSON has not generally been positive >> though. It's just not as human-readable. > > We're on the same line on that - hence the following proposal. I was > pretty sure switching the default format was a bad thing, but I had to > submit it in order to be complete ;). > Let's focus on the other one, as it's more friendly for everyone. > >> >> The other problem with changing log format defaults is that many people >> have already set up filters and processing based on the existing log >> format. There are significant user impacts to changing that default. >> >>> >>> - Provide a way to configure the output format/handler >>> While poking around in the puppet modules code, I didn't find any way to >>> set the output handler for the logs. For example, in puppet-nova³ we can >>> set a lot of things, but not the useful handler for the output. >>> >>> It would be really cool to get, for each puppet module, the capability >>> to set the handler so that one can just push some stuff in hiera, and >>> Voilà, we have JSON logs. >>> >>> Doing so would allow people to chose between the default (current) >>> output, and something more "computable". >> >> I think we should do this regardless. There are valid arguments for >> people to want both log formats IMHO and we should allow people to use >> what they want. > > If I understand the things correctly, that would require a "small" > change in every puppet modules so that it configures the service with > the proper log output. Any way to automate something on that? It might > be worth to do some PoC on a specific module maybe? >
So for the puppet modules, the available logging configuration lives in puppet-oslo so if there's a configuration around that that needs to be updated[0] then that is the first change. From there you would need to go through each module to expose this new configuration in the logging classes. For example puppet-nova[1]. You could probably write a script to modify each but I'm not sure they are completely consistent between classes. I think they are but it would require some validation. [0] https://github.com/openstack/puppet-oslo/blob/master/manifests/log.pp [1] https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp > >> >>> >>> Of course, either proposal will require a nice code change in all puppet >>> modules (add a new parameter for the foo::logging class, and use that >>> new param in the configuration file, and so on), but at least people >>> will be able to actually chose. >>> >>> So, before opening an issue on each launchpad project (that would be… >>> long), I'd rather open the discussion in here and, eventually, come to >>> some nice, acceptable and accepted solution that would make the >>> Openstack Community happy :). >>> >>> Any thoughts? >>> >>> Thank you for your attention, feedback and wonderful support for that >>> monster project :). >>> >>> Cheers, >>> >>> C. >>> >>> >>> ¹ >>> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235 >>> >>> ² >>> http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14 >>> >>> ³ >>> https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp >>> >>> >>> >>> >>> __________________________________________________________________________ >>> >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev