Here is why I am on board with this. As we have discovered, the logging with the syslog plugin leaves alot to be desired. It (to my understanding) still can't save tracebacks/stacktraces to the log files for whatever reason. stdout/stderr however works perfectly fine. That said the Docker log stuff has been a source of pain in the past, but it has gotten better. It does have the limitation of being only able to log one output at a time. This means, as an example, the neutron-dhcp-agent could send its logs to stdout/err but the dnsmasq process that it launch (that also has logs) would have to mix its logs in with the neutron logs in stdout/err. Can Heka handle this and separate them efficiently? Otherwise I see no choice but to stick with something that can handle multiple logs from a single container.
Sam Yaple On Mon, Jan 11, 2016 at 10:16 PM, Eric LEMOINE <elemo...@mirantis.com> wrote: > > Le 11 janv. 2016 18:45, "Michał Jastrzębski" <inc...@gmail.com> a écrit : > > > > On 11 January 2016 at 10:55, Eric LEMOINE <elemo...@mirantis.com> wrote: > > > Currently the services running in containers send their logs to > > > rsyslog. And rsyslog stores the logs in local files, located in the > > > host's /var/log directory. > > > > Yeah, however plan was to teach rsyslog to forward logs to central > > logging stack once this thing is implemented. > > Yes. With the current ELK Change Request, Rsyslog sends logs to the > central Logstash instance. If you read my design doc you'll understand that > it's precisely what we're proposing changing. > > > > I know. Our plan is to rely on Docker. Basically: containers write > > > their logs to stdout. The logs are collected by Docker Engine, which > > > makes them available through the unix:///var/run/docker.sock socket. > > > The socket is mounted into the Heka container, which uses the Docker > > > Log Input plugin [*] to reads the logs from that that socket. > > > > > > [*] < > http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html> > > > > So docker logs isn't best thing there is, however I'd suspect that's > > mostly console output fault. If you can tap into stdout efficiently, > > I'd say that's pretty good option. > > I'm not following you. Could you please be more specific? > > > >> Seems to me we need additional comparason of heka vs rsyslog;) Also > > >> this would have to be hands down better because rsyslog is already > > >> implemented, working and most of operators knows how to use it. > > > > > > > > > We don't need to remove Rsyslog. Services running in containers can > > > write their logs to both Rsyslog and stdout, which even is what they > > > do today (at least for the OpenStack services). > > > > > > > There is no point for that imho. I don't want to have several systems > > doing the same thing. Let's make decision of one, but optimal toolset. > > Could you please describe bottoms up what would your logging stack > > look like? Heka listening on stdout, transfering stuff to > > elasticsearch and kibana on top of it? > > My plan is to provide details in the blueprint document, that I'll > continue working on if the core developers agree with the principles of the > proposed architecture and change. > > But here's our plan—as already described in my previous email: the Kolla > services, which run in containers, write their logs to stdout. Logs are > collected by the Docker engine. Heka's Docker Log Input plugin is used to > read the container logs from the Docker endpoint (Unix socket). Since Heka > will run in a container a volume is necessary for accessing the Docker > endpoint. The Docker Log Input plugin inserts the logs into the Heka > pipeline, at the end of which an Elasticsearch Output plugin will send the > log messages to Elasticsearch. Here's a blog post reporting on that > approach: < > http://www.ianneubert.com/wp/2015/03/03/how-to-use-heka-docker-and-tutum/>. > We haven't tested that approach yet, but we plan to experiment with it as > we work on the specs. > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev