Jeffrey, Logstash-forwarder is deprecated upstream, so we can’t rely on that. Elastic's replacement is filebeat.
I’m not sure which one meets the requirements – filebeat or fluentd. In kolla-kubernetes fluentd is being used, and is well maintained. Both implementations are pretty green IMO. Not sure if fluentd also does log processing. I think its crucial to pick a component that just does log forwarding since that is the part that was deprecated. Our system has no log stash at all in it, and I’d like to keep it that way. Logstash is unnecessary for our use case. What we want is forwarder->es->cabana. Whatever forwarder is chosen, recommend picking the best of the two choices. I’d start with defining best as “does it solve the same problem as Heka does in our current implementation” then sprinkle throughput and minimal cpu and network utilization on top. If we can’t make a decision from there, not sure I have any further suggestions as I am not writing the code. Regards -steve From: Jeffrey Zhang <zhang.lei....@gmail.com<mailto:zhang.lei....@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Sunday, November 27, 2016 at 9:40 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [kolla] the alternative of log processing tool So filebeat is working with Logstash right? We need split the logs into pieces by using logstash. IMU, Filebeat do not a variety of processing plugins, like Logstash[0]. [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <sigmaviru...@gmail.com<mailto:sigmaviru...@gmail.com>> wrote: File beat is maintained be elastic and a part of their product line just like ELK. It's a fantastic tool and quite flexible given its age and size of codebase On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <zhang.lei....@gmail.com<mailto:zhang.lei....@gmail.com>> wrote: Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat. For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent to replace the Logstash at all. > Filebeat is based on the Logstash Forwarder source code and replaces Logstash > Forwarder as the method to use for tailing log files and forwarding them to > Logstash. Fillebeat is a log transport tool rather than log processing too. I do not treat it as an alternative at all. To be honest, I'd like back to Logstash, and Logstash 5.x is released with high performance improvement[4]. > In our performance testing, we've seen consistent throughput increases > across multiple configurations. In some cases, we observed up to 75% > increase in events processed through Logstash. another benefit to using Logstash is the whole ELK stack is maintained by one community/company. It is well tested and easy to upgrade the whole stack at the same time. Using other tools may force us on certain elasticsearch release. So, I think we have to alternative tools. * Fluentd * Logstash IMO, we need to make the decision and at least prepare the migration solution now. [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation [2] https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html [3] http://www.fluentd.org/ [4] https://www.elastic.co/blog/logstash-5-0-0-released -- Regards, Jeffrey Zhang Blog: http://xcodest.me<http://xcodest.me/> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Regards, Jeffrey Zhang Blog: http://xcodest.me<http://xcodest.me/>
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
