Re: [openstack-dev] [kolla] the alternative of log processing tool

On Tue, Nov 29, 2016 at 12:47 PM, Joshua Harlow wrote: > Why are people/things parsing tracebacks out of log files when the > following exists: > > http://docs.openstack.org/developer/oslo.log/api/formatters. > html#oslo_log.formatters.JSONFormatter > > Seems like oslo.log

Re: [openstack-dev] [kolla] the alternative of log processing tool

On Tue, Nov 29, 2016 at 11:28 AM, Michał Jastrzębski wrote: > Having custom /dev/log was real pain in few occasions. Also syslog was > particularly bad in working with multi-line logging (like python > tracebacks). > Heka reads local log files, makes things easier, and parses

Re: [openstack-dev] [kolla] the alternative of log processing tool

Why are people/things parsing tracebacks out of log files when the following exists: http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.JSONFormatter Seems like oslo.log also has a fluent formatter @

Re: [openstack-dev] [kolla] the alternative of log processing tool

Having custom /dev/log was real pain in few occasions. Also syslog was particularly bad in working with multi-line logging (like python tracebacks). Heka reads local log files, makes things easier, and parses things like tracebacks in it. It's my understanding that fluentd can do the same. On 28

Re: [openstack-dev] [kolla] the alternative of log processing tool

On Tue, Nov 29, 2016 at 7:29 AM, Michał Jastrzębski wrote: > I don't really like logstash as it's big memory eating beast. We had > good arch without it, and I'd like to keep it this way. Even with > logstash we still would need to use rsyslog to push logs around to > logstash,

Re: [openstack-dev] [kolla] the alternative of log processing tool

I don't really like logstash as it's big memory eating beast. We had good arch without it, and I'd like to keep it this way. Even with logstash we still would need to use rsyslog to push logs around to logstash, and that's a pita (trust me, I wrote it.). Fluentd just became cncf-backed project so

Re: [openstack-dev] [kolla] the alternative of log processing tool

> On 27 Nov 2016, at 06:55, Jeffrey Zhang wrote: > > * Fluentd > * Logstash I do not have a strong behaviour. At the moment we use the E and K of the ELK stack. Because of that I think it makes sense to go with Logstash. In this way, we have a stack developed by one

Re: [openstack-dev] [kolla] the alternative of log processing tool

e only positive feedback. > > > > Serguei > > > > *From:* Steven Dake (stdake) > *Sent:* Sunday, November 27, 2016 3:46 PM > > *To:* OpenStack Development Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > *Subject:* Re: [openstack-

Re: [openstack-dev] [kolla] the alternative of log processing tool

rds > -steve > > > From: Jeffrey Zhang <zhang.lei@gmail.com> > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Date: Sunday, November 27, 2016 at 9:40 AM > To: "OpenStack Develop

Re: [openstack-dev] [kolla] the alternative of log processing tool

ubject: Re: [openstack-dev] [kolla] the alternative of log processing tool Jeffrey, Logstash-forwarder is deprecated upstream, so we can’t rely on that. Elastic's replacement is filebeat. I’m not sure which one meets the requirements – filebeat or fluentd. In kolla-kubernetes fluentd is bein

Re: [openstack-dev] [kolla] the alternative of log processing tool

tions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Sunday, November 27, 2016 at 9:40 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>&g

Re: [openstack-dev] [kolla] the alternative of log processing tool

After reading the docs in github, i do not think Snap can handle logs very well. Snap introduce itself as "The open telemetry framework". I think it is more like ceilometer/collectd/zabbix. I also can not find how to create a mock syslog socket to collect logs[0], how to use regexp to split the

Re: [openstack-dev] [kolla] the alternative of log processing tool

So filebeat is working with Logstash right? We need split the logs into pieces by using logstash. IMU, Filebeat do not a variety of processing plugins, like Logstash[0]. [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco

Re: [openstack-dev] [kolla] the alternative of log processing tool

Thanks for quick reply. so I think the snap is not good choice。 For end user, hope the solustion is mature and easy maintain. as discuss before, I'll put on my operator hat and would like to give my +1 to keep ELK instead of Heka.

Re: [openstack-dev] [kolla] the alternative of log processing tool

File beat is maintained be elastic and a part of their product line just like ELK. It's a fantastic tool and quite flexible given its age and size of codebase On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" wrote: > Heka is marked deprecated in Kolla during Newton cycle[0].

Re: [openstack-dev] [kolla] the alternative of log processing tool

It's in development On 27 November 2016 at 08:50, Shake Chen wrote: > Hi Michal > > the snap seem not support log forward now, I can not find any infomation in > google . > > On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski > wrote: >> >> Hey, >> >> I

Re: [openstack-dev] [kolla] the alternative of log processing tool

Hi Michal the snap seem not support log forward now, I can not find any infomation in google . On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski wrote: > Hey, > > I am also working with Snap community to enable log forwarding with it [1]. > Snap is super lightweight and

Re: [openstack-dev] [kolla] the alternative of log processing tool

Hey, I am also working with Snap community to enable log forwarding with it [1]. Snap is super lightweight and additional benefit of this solution would be that it can also handle monitoring, which was it's initial role. One service to handle both would be elegant. I'll keep you posted but let's

[openstack-dev] [kolla] the alternative of log processing tool

Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat. For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent to replace the Logstash at all. > Filebeat is based on the Logstash