So tracebacks sort of works, they're there just ugly. That's why I'm also happy if we change rsyslog to heka.
Eric, I hope I wont ask too much, but could you please prepare PoC of kolla+heka, for what I care heka can log to local log file same as rsyslog for now. Would that be big problem? On 11 January 2016 at 16:37, Sam Yaple <sam...@yaple.net> wrote: > Here is why I am on board with this. As we have discovered, the logging with > the syslog plugin leaves alot to be desired. It (to my understanding) still > can't save tracebacks/stacktraces to the log files for whatever reason. > stdout/stderr however works perfectly fine. That said the Docker log stuff > has been a source of pain in the past, but it has gotten better. It does > have the limitation of being only able to log one output at a time. This > means, as an example, the neutron-dhcp-agent could send its logs to > stdout/err but the dnsmasq process that it launch (that also has logs) would > have to mix its logs in with the neutron logs in stdout/err. Can Heka handle > this and separate them efficiently? Otherwise I see no choice but to stick > with something that can handle multiple logs from a single container. > > Sam Yaple > > On Mon, Jan 11, 2016 at 10:16 PM, Eric LEMOINE <elemo...@mirantis.com> > wrote: >> >> >> Le 11 janv. 2016 18:45, "Michał Jastrzębski" <inc...@gmail.com> a écrit : >> > >> > On 11 January 2016 at 10:55, Eric LEMOINE <elemo...@mirantis.com> wrote: >> > > Currently the services running in containers send their logs to >> > > rsyslog. And rsyslog stores the logs in local files, located in the >> > > host's /var/log directory. >> > >> > Yeah, however plan was to teach rsyslog to forward logs to central >> > logging stack once this thing is implemented. >> >> Yes. With the current ELK Change Request, Rsyslog sends logs to the >> central Logstash instance. If you read my design doc you'll understand that >> it's precisely what we're proposing changing. >> >> > > I know. Our plan is to rely on Docker. Basically: containers write >> > > their logs to stdout. The logs are collected by Docker Engine, which >> > > makes them available through the unix:///var/run/docker.sock socket. >> > > The socket is mounted into the Heka container, which uses the Docker >> > > Log Input plugin [*] to reads the logs from that that socket. >> > > >> > > [*] >> > > <http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html> >> > >> > So docker logs isn't best thing there is, however I'd suspect that's >> > mostly console output fault. If you can tap into stdout efficiently, >> > I'd say that's pretty good option. >> >> I'm not following you. Could you please be more specific? >> >> > >> Seems to me we need additional comparason of heka vs rsyslog;) Also >> > >> this would have to be hands down better because rsyslog is already >> > >> implemented, working and most of operators knows how to use it. >> > > >> > > >> > > We don't need to remove Rsyslog. Services running in containers can >> > > write their logs to both Rsyslog and stdout, which even is what they >> > > do today (at least for the OpenStack services). >> > > >> > >> > There is no point for that imho. I don't want to have several systems >> > doing the same thing. Let's make decision of one, but optimal toolset. >> > Could you please describe bottoms up what would your logging stack >> > look like? Heka listening on stdout, transfering stuff to >> > elasticsearch and kibana on top of it? >> >> My plan is to provide details in the blueprint document, that I'll >> continue working on if the core developers agree with the principles of the >> proposed architecture and change. >> >> But here's our plan—as already described in my previous email: the Kolla >> services, which run in containers, write their logs to stdout. Logs are >> collected by the Docker engine. Heka's Docker Log Input plugin is used to >> read the container logs from the Docker endpoint (Unix socket). Since Heka >> will run in a container a volume is necessary for accessing the Docker >> endpoint. The Docker Log Input plugin inserts the logs into the Heka >> pipeline, at the end of which an Elasticsearch Output plugin will send the >> log messages to Elasticsearch. Here's a blog post reporting on that >> approach: >> <http://www.ianneubert.com/wp/2015/03/03/how-to-use-heka-docker-and-tutum/>. >> We haven't tested that approach yet, but we plan to experiment with it as we >> work on the specs. >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev