Just FYI. Docker support multi log driver now[0].
Version New Logging Drivers 1.6.0 json-file, syslog, none 1.7.0 journald 1.8.0 fluentd, gelf 1.9.0 awslogs For the stdout from container, we can use gelf/syslog driver to forward. [0] https://docs.docker.com/engine/reference/logging/overview/ On Fri, Jan 15, 2016 at 8:28 AM, Steven Dake (stdake) <std...@cisco.com> wrote: > Eric, > > Comments inline. > > On 1/14/16, 3:31 AM, "Eric LEMOINE" <elemo...@mirantis.com> wrote: > > >On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) <std...@cisco.com> > >wrote: > >> Eric, > > > > > >Hi Steven > > > Feel free to call me Steve > > > > > > >> > >> Apologies for top post, not really sure where in this thread to post > >>this > >> list of questions as its sort of a change in topic so I changed the > >> subject line :) > >> > >> 1. > >> Somewhere I read when researching this Heka topic, that Heka cannot log > >> all details from /dev/log. Some services like mariadb for example don't > >> log to stdout as I think Heka requires to operate correctly. Would you > >> mind responding on the question "Would Heka be able to effectively log > >> every piece of information coming off the system related to OpenStack > >>(our > >> infrastructure services like ceph/mariadb/etc as well as the OpenStack > >> services)? > > > > > >My first reaction to this is: if we have services, such as mariadb, > >that can only send their logs to syslog then let's continue using > >Rsyslog. And with Rsyslog we can easily store logs on the local > >filesystem as well (your requirement #3 below). > > Rsyslog may be the best tool for this job. I'll be looking to your POC to > get a feel for the validity of that claim :) > > > > >That being said, it appears that Heka supports reading logs from > >/dev/log. This can be done using the UdpInput plugin with "net" set > >to "unixgram". See > ><https://github.com/mozilla-services/heka/issues/790> for the original > >issue. Heka also supports writing logs to files on the local > >filesystem, through the FileOutput plugin. We do not currently use > >the UdpInput plugin, so we need to test it and see if it can work for > >Kolla. We will work on these tests, and report back to the list. > > > > > > > >> 2. > >> Also, I want to make sure we can fix up the backtrace defeciency. > >> Currently rsyslog doesn't log backtraces in python code. Perhaps Sam or > >> inc0 know the reason behind it, but I want to make sure we can fix up > >>this > >> annoyance, because backtraces are mightily important. > > > > > >I've had a look on my AIO Kolla. And I do see Python tracebacks in > >OpenStack log files created by Rsyslog (in > >/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example). > >They're just on a single line, with "#012" used as the separator [*]. > >So they are hard to read, but they are there. I think that is > >consistent with what SamYaple and inc0 said yesterday on IRC. > > > >[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive > >setting. See > >< > http://www.rsyslog.com/doc/master/configuration/input_directives/rsconf1_ > >escapecontrolcharactersonreceive.html>. > > > > Apparently the one-lining of the tracebacks is a new feature because of a > bug fix in oslo.log that dims fixed for us. So my original comments about > missing backtraces was dated information from December. > > > > >> 3. > >> Also I want to make sure each node ends up with log files in a data > >> container (or data volume or whatever we just recently replaced the data > >> containers with) for all the services for individual node diagnostics. > >> This helps fill the gap of the Kibana visualization and Elasticsearch > >> where we may not have a perfect diagnostic solution at the conclusion of > >> Mitaka and in need of individual node inspection of the logs. Can Heka > >>be > >> made to do this? Our rsyslog implementation does today, and its a hard > >> requirement for the moment. If we need some special software to run in > >> addition to Heka, I could live with that. > > > > > >That "special software" could be Rsyslog :) Seriously, Rsyslog > >provides a solution for getting logs from services that only log to > >syslog. We can also easily configure Rsyslog to write logs on the > >local filesystem, as done in Kolla already today. And using Heka we > >can easily make Python tracebacks look good in Kibana. > > > >I would like to point out that our initial intent was not to remove > >Rsyslog. Our intent was to propose a scalable/decentralized log > >processing architecture based on Heka running on each node, instead of > >relying on a centralized Logstash instance. Using Heka we eliminate > >the need to deploy and manage a resilient Logstash/Redis cluster. And > >it is to be noted that Heka gives us a lot of flexibility. In > >particular, Heka makes it possible to collect logs from services that > >don't speak syslog (RabbitMQ for example, whose logs are not currently > >collected!). > > > FWIW the concern that logstash isn't scalable is unsubstantiated. IMNSHO > the reason to use Heka is it does two things we need with one component > while removing JVMs from each compute node. This is my main attraction to > the adoption of Heka. > > > > >As mentioned above Heka provides plugins that we could possibly > >leverage to remove Rsyslog completely, but at this point we cannot > >guarantee that they will do the job. Our coming tests will tell. > > If the plugins work in a way in which we can remove rsyslog completely, I > would prefer that outcome. Less dependencies in software systems are > better. > > Regards > -steve > > > > >Thanks. > > > >__________________________________________________________________________ > >OpenStack Development Mailing List (not for usage questions) > >Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Jeffrey Zhang Blog: http://xcodest.me twitter/weibo: @jeffrey4l
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev