subject:"Re\: \[openstack\-dev\] \[kolla\] the alternative of log processing tool"

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Jeffrey Zhang

On Tue, Nov 29, 2016 at 12:47 PM, Joshua Harlow 
wrote:

> Why are people/things parsing tracebacks out of log files when the
> following exists:
>
> http://docs.openstack.org/developer/oslo.log/api/formatters.
> html#oslo_log.formatters.JSONFormatter
>
> Seems like oslo.log also has a fluent formatter @
> http://docs.openstack.org/developer/oslo.log/api/formatters.
> html#oslo_log.formatters.FluentFormatter.
>
> IMHO I hope that nobody is making regex or such for tracebacks any more in
> the 21st century; structured data ftw...
>

Nice information, harlowja. It will be helpful when moving to fluentd.



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Jeffrey Zhang

On Tue, Nov 29, 2016 at 11:28 AM, Michał Jastrzębski 
wrote:

> Having custom /dev/log was real pain in few occasions. Also syslog was
> particularly bad in working with multi-line logging (like python
> tracebacks).
> Heka reads local log files, makes things easier, and parses things
> like tracebacks in it. It's my understanding that fluentd can do the
> same.
>

In some time, the syslog is the only choice, for example, the swift and
keepalived case.

btw, kolla is using /dev/log created by heka now.



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Joshua Harlow

Why are people/things parsing tracebacks out of log files when the
following exists:

http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.JSONFormatter

Seems like oslo.log also has a fluent formatter @
http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.FluentFormatter.

IMHO I hope that nobody is making regex or such for tracebacks any more
in the 21st century; structured data ftw...

Michał Jastrzębski wrote:

Having custom /dev/log was real pain in few occasions. Also syslog was
particularly bad in working with multi-line logging (like python
tracebacks).
Heka reads local log files, makes things easier, and parses things
like tracebacks in it. It's my understanding that fluentd can do the
same.

On 28 November 2016 at 19:07, Jeffrey Zhang wrote:

On Tue, Nov 29, 2016 at 7:29 AM, Michał Jastrzębski
wrote:

I don't really like logstash as it's big memory eating beast. We had
good arch without it, and I'd like to keep it this way. Even with
logstash we still would need to use rsyslog to push logs around to
logstash, and that's a pita (trust me, I wrote it.).

About the rsyslog mock socket, i think we have fixed it.
heka is using rsyslog socket too. IIUC, i know the issue. could u explain
more?

Fluentd just became cncf-backed project so that's a plus for fluentd.
Out of fliebeat+logstash vs flutentd I'd go with fluentd.

So, I am OK to use fluentd.

;)

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Michał Jastrzębski

Having custom /dev/log was real pain in few occasions. Also syslog was
particularly bad in working with multi-line logging (like python
tracebacks).
Heka reads local log files, makes things easier, and parses things
like tracebacks in it. It's my understanding that fluentd can do the
same.

On 28 November 2016 at 19:07, Jeffrey Zhang  wrote:
>
> On Tue, Nov 29, 2016 at 7:29 AM, Michał Jastrzębski 
> wrote:
>>
>> I don't really like logstash as it's big memory eating beast. We had
>> good arch without it, and I'd like to keep it this way. Even with
>> logstash we still would need to use rsyslog to push logs around to
>> logstash, and that's a pita (trust me, I wrote it.).
>
> About the rsyslog mock socket, i think we have fixed it.
> heka is using rsyslog socket too. IIUC, i know the issue. could u explain
> more?
>>
>>
>> Fluentd just became cncf-backed project so that's a plus for fluentd.
>> Out of fliebeat+logstash vs flutentd I'd go with fluentd.
>
>
> So, I am OK to use fluentd.
>
> ;)
>
>
>
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Jeffrey Zhang

On Tue, Nov 29, 2016 at 7:29 AM, Michał Jastrzębski 
wrote:

> I don't really like logstash as it's big memory eating beast. We had
> good arch without it, and I'd like to keep it this way. Even with
> logstash we still would need to use rsyslog to push logs around to
> logstash, and that's a pita (trust me, I wrote it.).
>
About the rsyslog mock socket, i think we have fixed it.
heka is using rsyslog socket too. IIUC, i know the issue. could u explain
more?

>
> Fluentd just became cncf-backed project so that's a plus for fluentd.
> Out of fliebeat+logstash vs flutentd I'd go with fluentd.
>

So, I am OK to use fluentd.

;)




-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Michał Jastrzębski

I don't really like logstash as it's big memory eating beast. We had
good arch without it, and I'd like to keep it this way. Even with
logstash we still would need to use rsyslog to push logs around to
logstash, and that's a pita (trust me, I wrote it.).

Fluentd just became cncf-backed project so that's a plus for fluentd.
Out of fliebeat+logstash vs flutentd I'd go with fluentd.

On 28 November 2016 at 03:15, Christian Berendt
 wrote:
>
>> On 27 Nov 2016, at 06:55, Jeffrey Zhang  wrote:
>>
>> * Fluentd
>> * Logstash
>
> I do not have a strong behaviour.
>
> At the moment we use the E and K of the ELK stack. Because of that I think it 
> makes sense to go with Logstash. In this way, we have a stack developed by 
> one community and which is tuned to each other.
>
> Christian.
>
> --
> Christian Berendt
> Chief Executive Officer (CEO)
>
> Mail: bere...@betacloud-solutions.de
> Web: https://www.betacloud-solutions.de
>
> Betacloud Solutions GmbH
> Teckstrasse 62 / 70190 Stuttgart / Deutschland
>
> Geschäftsführer: Christian Berendt
> Unternehmenssitz: Stuttgart
> Amtsgericht: Stuttgart, HRB 756139
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-28 Thread Christian Berendt


> On 27 Nov 2016, at 06:55, Jeffrey Zhang  wrote:
> 
> * Fluentd
> * Logstash

I do not have a strong behaviour.

At the moment we use the E and K of the ELK stack. Because of that I think it 
makes sense to go with Logstash. In this way, we have a stack developed by one 
community and which is tuned to each other.

Christian.

-- 
Christian Berendt
Chief Executive Officer (CEO)

Mail: bere...@betacloud-solutions.de
Web: https://www.betacloud-solutions.de

Betacloud Solutions GmbH
Teckstrasse 62 / 70190 Stuttgart / Deutschland

Geschäftsführer: Christian Berendt
Unternehmenssitz: Stuttgart
Amtsgericht: Stuttgart, HRB 756139


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Jeffrey Zhang

sbezverk,

yep. Event though I prefer to use Logstash. But fluentd is OK to me.

On Mon, Nov 28, 2016 at 8:31 AM, Serguei Bezverkhi (sbezverk) <
sbezv...@cisco.com> wrote:

> I would vote for fluentd because: 1 – it has been around since 2011 so it
> is hard to call it green. 2 – There is constant development of new
> features/filters, 3 – As it was mentioned at Kubecon by fluentd people,
> they are deeply committed to Open Source community so rumors that they
> would go private does not sound reliable. We use it in kolla kubernetes and
> so far we could share only positive feedback.
>
>
>
> Serguei
>
>
>
> *From:* Steven Dake (stdake)
> *Sent:* Sunday, November 27, 2016 3:46 PM
>
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [kolla] the alternative of log processing
> tool
>
>
>
> Jeffrey,
>
>
>
> Logstash-forwarder is deprecated upstream, so we can’t rely on that.
> Elastic's replacement is filebeat.
>
>
>
> I’m not sure which one meets the requirements – filebeat or fluentd.  In
> kolla-kubernetes fluentd is being used, and is well maintained.  Both
> implementations are pretty green IMO.  Not sure if fluentd also does log
> processing.  I think its crucial to pick a component that just does log
> forwarding since that is the part that was deprecated.
>
>
>
> Our system has no log stash at all in it, and I’d like to keep it that
> way.  Logstash is unnecessary for our use case.  What we want is
> forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the
> best of the two choices.  I’d start with defining best as “does it solve
> the same problem as Heka does in our current implementation” then sprinkle
> throughput and minimal cpu and network utilization on top.  If we can’t
> make a decision from there, not sure I have any further suggestions as I am
> not writing the code.
>
>
>
> Regards
>
> -steve
>
>
>
>
>
> *From: *Jeffrey Zhang <zhang.lei@gmail.com>
> *Reply-To: *"OpenStack Development Mailing List (not for usage
> questions)" <openstack-dev@lists.openstack.org>
> *Date: *Sunday, November 27, 2016 at 9:40 AM
> *To: *"OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> *Subject: *Re: [openstack-dev] [kolla] the alternative of log processing
> tool
>
>
>
> So filebeat is working with Logstash right? We need split the logs into
> pieces by using logstash. IMU, Filebeat do not a variety of processing
> plugins, like Logstash[0].
>
>
>
> [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
>
>
>
> On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <sigmaviru...@gmail.com>
> wrote:
>
> File beat is maintained be elastic and a part of their product line just
> like ELK. It's a fantastic tool and quite flexible given its age and size
> of codebase
>
>
>
> On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <zhang.lei@gmail.com> wrote:
>
> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have
> a
>
> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>
>
>
> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> intent
>
> to replace the Logstash at all.
>
>
>
> > Filebeat is based on the Logstash Forwarder source code and replaces
> Logstash
>
> > Forwarder as the method to use for tailing log files and forwarding them
> to
>
> > Logstash.
>
>
>
> Fillebeat is a log transport tool rather than log processing too. I do not
>
> treat it as an alternative at all.
>
>
>
> To be honest, I'd like back to Logstash, and Logstash 5.x is released with
> high
>
> performance improvement[4].
>
>
>
> >  In our performance testing, we've seen consistent throughput increases
>
> >  across multiple configurations. In some cases, we observed up to 75%
>
> >  increase in events processed through Logstash.
>
>
>
> another benefit to using Logstash is the whole ELK stack is maintained by
> one
>
> community/company. It is well tested and easy to upgrade the whole stack
> at the
>
> same time. Using other tools may force us on certain elasticsearch release.
>
>
>
> So, I think we have to alternative tools.
>
>
>
> * Fluentd
>
> * Logstash
>
>
>
> IMO, we need to make the decision and at least prepare the migration
> solution now.
>
>
>
> [1] https://blueprints.launchpad.net/koll

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Jeffrey Zhang

Logstash-forward/Filebeat just cut logs in preparation for processing
elsewhere. It doesn't process logs just forward it to another processor (
Logstash / Heka / Fluentd ). It do not have any processing filter like
Logstash. At least, we need some thing tool like grok, syslog intput etc.

what we need is:

* listen on syslog like socket to collect logs
* processing plugin, like logstash grok does.

I do not think fielbeat meet this requirement. So finally, we need

 -> filebeat ( maybe, log forward ) -> Logstash/heka/Fluentd ( log
processing ) -> ES ( log storage ) -> grafana ( log ui )



On Mon, Nov 28, 2016 at 4:45 AM, Steven Dake (stdake) <std...@cisco.com>
wrote:

> Jeffrey,
>
> Logstash-forwarder is deprecated upstream, so we can’t rely on that.
> Elastic's replacement is filebeat.
>
> I’m not sure which one meets the requirements – filebeat or fluentd.  In
> kolla-kubernetes fluentd is being used, and is well maintained.  Both
> implementations are pretty green IMO.  Not sure if fluentd also does log
> processing.  I think its crucial to pick a component that just does log
> forwarding since that is the part that was deprecated.
>
> Our system has no log stash at all in it, and I’d like to keep it that
> way.  Logstash is unnecessary for our use case.  What we want is
> forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the
> best of the two choices.  I’d start with defining best as “does it solve
> the same problem as Heka does in our current implementation” then sprinkle
> throughput and minimal cpu and network utilization on top.  If we can’t
> make a decision from there, not sure I have any further suggestions as I am
> not writing the code.
>
> Regards
> -steve
>
>
> From: Jeffrey Zhang <zhang.lei@gmail.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Sunday, November 27, 2016 at 9:40 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [kolla] the alternative of log processing
> tool
>
> So filebeat is working with Logstash right? We need split the logs into
> pieces by using logstash. IMU, Filebeat do not a variety of processing
> plugins, like Logstash[0].
>
> [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
>
> On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <sigmaviru...@gmail.com>
> wrote:
>
>> File beat is maintained be elastic and a part of their product line just
>> like ELK. It's a fantastic tool and quite flexible given its age and size
>> of codebase
>>
>> On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <zhang.lei@gmail.com>
>> wrote:
>>
>>> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
>>> have a
>>> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>>>
>>> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
>>> intent
>>> to replace the Logstash at all.
>>>
>>> > Filebeat is based on the Logstash Forwarder source code and replaces
>>> Logstash
>>> > Forwarder as the method to use for tailing log files and forwarding
>>> them to
>>> > Logstash.
>>>
>>> Fillebeat is a log transport tool rather than log processing too. I do
>>> not
>>> treat it as an alternative at all.
>>>
>>> To be honest, I'd like back to Logstash, and Logstash 5.x is released
>>> with high
>>> performance improvement[4].
>>>
>>> >  In our performance testing, we've seen consistent throughput increases
>>> >  across multiple configurations. In some cases, we observed up to 75%
>>> >  increase in events processed through Logstash.
>>>
>>> another benefit to using Logstash is the whole ELK stack is maintained
>>> by one
>>> community/company. It is well tested and easy to upgrade the whole stack
>>> at the
>>> same time. Using other tools may force us on certain elasticsearch
>>> release.
>>>
>>> So, I think we have to alternative tools.
>>>
>>> * Fluentd
>>> * Logstash
>>>
>>> IMO, we need to make the decision and at least prepare the migration
>>> solution now.
>>>
>>> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
>>> [2] https://www.elastic.co/guide/en/beats/filebeat/current/migra
>>> ting-from-logstash-forwarder.html
>>> [3] http://www.fluentd.org/
>&

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Serguei Bezverkhi (sbezverk)

I would vote for fluentd because: 1 – it has been around since 2011 so it is 
hard to call it green. 2 – There is constant development of new 
features/filters, 3 – As it was mentioned at Kubecon by fluentd people, they 
are deeply committed to Open Source community so rumors that they would go 
private does not sound reliable. We use it in kolla kubernetes and so far we 
could share only positive feedback.

Serguei

From: Steven Dake (stdake)
Sent: Sunday, November 27, 2016 3:46 PM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [kolla] the alternative of log processing tool

Jeffrey,

Logstash-forwarder is deprecated upstream, so we can’t rely on that.  Elastic's 
replacement is filebeat.

I’m not sure which one meets the requirements – filebeat or fluentd.  In 
kolla-kubernetes fluentd is being used, and is well maintained.  Both 
implementations are pretty green IMO.  Not sure if fluentd also does log 
processing.  I think its crucial to pick a component that just does log 
forwarding since that is the part that was deprecated.

Our system has no log stash at all in it, and I’d like to keep it that way.  
Logstash is unnecessary for our use case.  What we want is 
forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the 
best of the two choices.  I’d start with defining best as “does it solve the 
same problem as Heka does in our current implementation” then sprinkle 
throughput and minimal cpu and network utilization on top.  If we can’t make a 
decision from there, not sure I have any further suggestions as I am not 
writing the code.

Regards
-steve

From: Jeffrey Zhang <zhang.lei@gmail.com<mailto:zhang.lei@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, November 27, 2016 at 9:40 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kolla] the alternative of log processing tool

So filebeat is working with Logstash right? We need split the logs into pieces 
by using logstash. IMU, Filebeat do not a variety of processing plugins, like 
Logstash[0].

[0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco 
<sigmaviru...@gmail.com<mailto:sigmaviru...@gmail.com>> wrote:

File beat is maintained be elastic and a part of their product line just like 
ELK. It's a fantastic tool and quite flexible given its age and size of codebase

On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" 
<zhang.lei@gmail.com<mailto:zhang.lei@gmail.com>> wrote:
Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a
blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.

For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent
to replace the Logstash at all.

> Filebeat is based on the Logstash Forwarder source code and replaces Logstash
> Forwarder as the method to use for tailing log files and forwarding them to
> Logstash.

Fillebeat is a log transport tool rather than log processing too. I do not
treat it as an alternative at all.

To be honest, I'd like back to Logstash, and Logstash 5.x is released with high
performance improvement[4].

>  In our performance testing, we've seen consistent throughput increases
>  across multiple configurations. In some cases, we observed up to 75%
>  increase in events processed through Logstash.

another benefit to using Logstash is the whole ELK stack is maintained by one
community/company. It is well tested and easy to upgrade the whole stack at the
same time. Using other tools may force us on certain elasticsearch release.

So, I think we have to alternative tools.

* Fluentd
* Logstash

IMO, we need to make the decision and at least prepare the migration solution 
now.

[1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
[2] 
https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html
[3] http://www.fluentd.org/
[4] https://www.elastic.co/blog/logstash-5-0-0-released

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me<http://xcodest.me/>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
open

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Steven Dake (stdake)

Jeffrey,

Logstash-forwarder is deprecated upstream, so we can’t rely on that.  Elastic's 
replacement is filebeat.

I’m not sure which one meets the requirements – filebeat or fluentd.  In 
kolla-kubernetes fluentd is being used, and is well maintained.  Both 
implementations are pretty green IMO.  Not sure if fluentd also does log 
processing.  I think its crucial to pick a component that just does log 
forwarding since that is the part that was deprecated.

Our system has no log stash at all in it, and I’d like to keep it that way.  
Logstash is unnecessary for our use case.  What we want is 
forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the 
best of the two choices.  I’d start with defining best as “does it solve the 
same problem as Heka does in our current implementation” then sprinkle 
throughput and minimal cpu and network utilization on top.  If we can’t make a 
decision from there, not sure I have any further suggestions as I am not 
writing the code.

Regards
-steve

From: Jeffrey Zhang <zhang.lei@gmail.com<mailto:zhang.lei@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, November 27, 2016 at 9:40 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kolla] the alternative of log processing tool

So filebeat is working with Logstash right? We need split the logs into pieces 
by using logstash. IMU, Filebeat do not a variety of processing plugins, like 
Logstash[0].

[0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco 
<sigmaviru...@gmail.com<mailto:sigmaviru...@gmail.com>> wrote:

File beat is maintained be elastic and a part of their product line just like 
ELK. It's a fantastic tool and quite flexible given its age and size of codebase

On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" 
<zhang.lei@gmail.com<mailto:zhang.lei@gmail.com>> wrote:
Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a
blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.

For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent
to replace the Logstash at all.

> Filebeat is based on the Logstash Forwarder source code and replaces Logstash
> Forwarder as the method to use for tailing log files and forwarding them to
> Logstash.

Fillebeat is a log transport tool rather than log processing too. I do not
treat it as an alternative at all.

To be honest, I'd like back to Logstash, and Logstash 5.x is released with high
performance improvement[4].

>  In our performance testing, we've seen consistent throughput increases
>  across multiple configurations. In some cases, we observed up to 75%
>  increase in events processed through Logstash.

another benefit to using Logstash is the whole ELK stack is maintained by one
community/company. It is well tested and easy to upgrade the whole stack at the
same time. Using other tools may force us on certain elasticsearch release.

So, I think we have to alternative tools.

* Fluentd
* Logstash

IMO, we need to make the decision and at least prepare the migration solution 
now.

[1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
[2] 
https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html
[3] http://www.fluentd.org/
[4] https://www.elastic.co/blog/logstash-5-0-0-released

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me<http://xcodest.me/>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me<http://xcodest.me/>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Jeffrey Zhang

After reading the docs in github, i do not think Snap can handle logs very
well.  Snap introduce itself as "The open telemetry framework". I think it
is
more like ceilometer/collectd/zabbix. I also can not find how to create a
mock
syslog socket to collect logs[0], how to use regexp to split the logs?.
I do
not think we have enough time and energy to waiting until Snap fix it.



It may be a cool tool. But it is not mature, imo. Why not use a well used
and
mature software, even though it may be slower than others? But it works very
well and lots of guys are familiar with it.




[0] https://github.com/intelsdi-x/snap/issues/1117

On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski 
wrote:

> Hey,
>
> I am also working with Snap community to enable log forwarding with it [1].
> Snap is super lightweight and additional benefit of this solution
> would be that it can also handle monitoring, which was it's initial
> role. One service to handle both would be elegant. I'll keep you
> posted but let's not throw away this idea just yet.
>
> Cheers,
> Michal
>
> [1] https://github.com/intelsdi-x/snap
>
> On 26 November 2016 at 23:55, Jeffrey Zhang 
> wrote:
> > Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
> have a
> > blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
> >
> > For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> > intent
> > to replace the Logstash at all.
> >
> >> Filebeat is based on the Logstash Forwarder source code and replaces
> >> Logstash
> >> Forwarder as the method to use for tailing log files and forwarding them
> >> to
> >> Logstash.
> >
> > Fillebeat is a log transport tool rather than log processing too. I do
> not
> > treat it as an alternative at all.
> >
> > To be honest, I'd like back to Logstash, and Logstash 5.x is released
> with
> > high
> > performance improvement[4].
> >
> >>  In our performance testing, we've seen consistent throughput increases
> >>  across multiple configurations. In some cases, we observed up to 75%
> >>  increase in events processed through Logstash.
> >
> > another benefit to using Logstash is the whole ELK stack is maintained by
> > one
> > community/company. It is well tested and easy to upgrade the whole stack
> at
> > the
> > same time. Using other tools may force us on certain elasticsearch
> release.
> >
> > So, I think we have to alternative tools.
> >
> > * Fluentd
> > * Logstash
> >
> > IMO, we need to make the decision and at least prepare the migration
> > solution now.
> >
> > [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> > [2]
> > https://www.elastic.co/guide/en/beats/filebeat/current/
> migrating-from-logstash-forwarder.html
> > [3] http://www.fluentd.org/
> > [4] https://www.elastic.co/blog/logstash-5-0-0-released
> >
> > --
> > Regards,
> > Jeffrey Zhang
> > Blog: http://xcodest.me
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Jeffrey Zhang

So filebeat is working with Logstash right? We need split the logs into
pieces by using logstash. IMU, Filebeat do not a variety of processing
plugins, like Logstash[0].

[0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco 
wrote:

> File beat is maintained be elastic and a part of their product line just
> like ELK. It's a fantastic tool and quite flexible given its age and size
> of codebase
>
> On Nov 26, 2016 11:59 PM, "Jeffrey Zhang"  wrote:
>
>> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
>> have a
>> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>>
>> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
>> intent
>> to replace the Logstash at all.
>>
>> > Filebeat is based on the Logstash Forwarder source code and replaces
>> Logstash
>> > Forwarder as the method to use for tailing log files and forwarding
>> them to
>> > Logstash.
>>
>> Fillebeat is a log transport tool rather than log processing too. I do not
>> treat it as an alternative at all.
>>
>> To be honest, I'd like back to Logstash, and Logstash 5.x is released
>> with high
>> performance improvement[4].
>>
>> >  In our performance testing, we've seen consistent throughput increases
>> >  across multiple configurations. In some cases, we observed up to 75%
>> >  increase in events processed through Logstash.
>>
>> another benefit to using Logstash is the whole ELK stack is maintained by
>> one
>> community/company. It is well tested and easy to upgrade the whole stack
>> at the
>> same time. Using other tools may force us on certain elasticsearch
>> release.
>>
>> So, I think we have to alternative tools.
>>
>> * Fluentd
>> * Logstash
>>
>> IMO, we need to make the decision and at least prepare the migration
>> solution now.
>>
>> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
>> [2] https://www.elastic.co/guide/en/beats/filebeat/current/migra
>> ting-from-logstash-forwarder.html
>> [3] http://www.fluentd.org/
>> [4] https://www.elastic.co/blog/logstash-5-0-0-released
>>
>> --
>> Regards,
>> Jeffrey Zhang
>> Blog: http://xcodest.me
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Shake Chen

Thanks for quick reply.

so I think the snap is not good choice。

For end user, hope the solustion is mature and easy maintain.

as discuss before,

I'll put on my operator hat and would like to give my +1 to keep ELK instead
of Heka.


http://lists.openstack.org/pipermail/openstack-dev/2016-January/083974.html






On Sun, Nov 27, 2016 at 11:09 PM, Michał Jastrzębski 
wrote:

> It's in development
>
> On 27 November 2016 at 08:50, Shake Chen  wrote:
> > Hi Michal
> >
> > the snap seem not support log forward now, I can not find any infomation
> in
> > google .
> >
> > On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski 
> > wrote:
> >>
> >> Hey,
> >>
> >> I am also working with Snap community to enable log forwarding with it
> >> [1].
> >> Snap is super lightweight and additional benefit of this solution
> >> would be that it can also handle monitoring, which was it's initial
> >> role. One service to handle both would be elegant. I'll keep you
> >> posted but let's not throw away this idea just yet.
> >>
> >> Cheers,
> >> Michal
> >>
> >> [1] https://github.com/intelsdi-x/snap
> >>
> >> On 26 November 2016 at 23:55, Jeffrey Zhang 
> >> wrote:
> >> > Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
> >> > have a
> >> > blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
> >> >
> >> > For Filebeat, it is just a replacement of logstash-forward[2]. It is
> not
> >> > intent
> >> > to replace the Logstash at all.
> >> >
> >> >> Filebeat is based on the Logstash Forwarder source code and replaces
> >> >> Logstash
> >> >> Forwarder as the method to use for tailing log files and forwarding
> >> >> them
> >> >> to
> >> >> Logstash.
> >> >
> >> > Fillebeat is a log transport tool rather than log processing too. I do
> >> > not
> >> > treat it as an alternative at all.
> >> >
> >> > To be honest, I'd like back to Logstash, and Logstash 5.x is released
> >> > with
> >> > high
> >> > performance improvement[4].
> >> >
> >> >>  In our performance testing, we've seen consistent throughput
> increases
> >> >>  across multiple configurations. In some cases, we observed up to 75%
> >> >>  increase in events processed through Logstash.
> >> >
> >> > another benefit to using Logstash is the whole ELK stack is maintained
> >> > by
> >> > one
> >> > community/company. It is well tested and easy to upgrade the whole
> stack
> >> > at
> >> > the
> >> > same time. Using other tools may force us on certain elasticsearch
> >> > release.
> >> >
> >> > So, I think we have to alternative tools.
> >> >
> >> > * Fluentd
> >> > * Logstash
> >> >
> >> > IMO, we need to make the decision and at least prepare the migration
> >> > solution now.
> >> >
> >> > [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> >> > [2]
> >> >
> >> > https://www.elastic.co/guide/en/beats/filebeat/current/
> migrating-from-logstash-forwarder.html
> >> > [3] http://www.fluentd.org/
> >> > [4] https://www.elastic.co/blog/logstash-5-0-0-released
> >> >
> >> > --
> >> > Regards,
> >> > Jeffrey Zhang
> >> > Blog: http://xcodest.me
> >> >
> >> >
> >> > 
> __
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >> 
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > --
> > Shake Chen
> >
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Shake Chen
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Ian Cordasco

File beat is maintained be elastic and a part of their product line just
like ELK. It's a fantastic tool and quite flexible given its age and size
of codebase

On Nov 26, 2016 11:59 PM, "Jeffrey Zhang"  wrote:

> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have
> a
> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>
> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> intent
> to replace the Logstash at all.
>
> > Filebeat is based on the Logstash Forwarder source code and replaces
> Logstash
> > Forwarder as the method to use for tailing log files and forwarding them
> to
> > Logstash.
>
> Fillebeat is a log transport tool rather than log processing too. I do not
> treat it as an alternative at all.
>
> To be honest, I'd like back to Logstash, and Logstash 5.x is released with
> high
> performance improvement[4].
>
> >  In our performance testing, we've seen consistent throughput increases
> >  across multiple configurations. In some cases, we observed up to 75%
> >  increase in events processed through Logstash.
>
> another benefit to using Logstash is the whole ELK stack is maintained by
> one
> community/company. It is well tested and easy to upgrade the whole stack
> at the
> same time. Using other tools may force us on certain elasticsearch release.
>
> So, I think we have to alternative tools.
>
> * Fluentd
> * Logstash
>
> IMO, we need to make the decision and at least prepare the migration
> solution now.
>
> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> [2] https://www.elastic.co/guide/en/beats/filebeat/current/
> migrating-from-logstash-forwarder.html
> [3] http://www.fluentd.org/
> [4] https://www.elastic.co/blog/logstash-5-0-0-released
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Michał Jastrzębski

It's in development

On 27 November 2016 at 08:50, Shake Chen  wrote:
> Hi Michal
>
> the snap seem not support log forward now, I can not find any infomation in
> google .
>
> On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski 
> wrote:
>>
>> Hey,
>>
>> I am also working with Snap community to enable log forwarding with it
>> [1].
>> Snap is super lightweight and additional benefit of this solution
>> would be that it can also handle monitoring, which was it's initial
>> role. One service to handle both would be elegant. I'll keep you
>> posted but let's not throw away this idea just yet.
>>
>> Cheers,
>> Michal
>>
>> [1] https://github.com/intelsdi-x/snap
>>
>> On 26 November 2016 at 23:55, Jeffrey Zhang 
>> wrote:
>> > Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
>> > have a
>> > blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>> >
>> > For Filebeat, it is just a replacement of logstash-forward[2]. It is not
>> > intent
>> > to replace the Logstash at all.
>> >
>> >> Filebeat is based on the Logstash Forwarder source code and replaces
>> >> Logstash
>> >> Forwarder as the method to use for tailing log files and forwarding
>> >> them
>> >> to
>> >> Logstash.
>> >
>> > Fillebeat is a log transport tool rather than log processing too. I do
>> > not
>> > treat it as an alternative at all.
>> >
>> > To be honest, I'd like back to Logstash, and Logstash 5.x is released
>> > with
>> > high
>> > performance improvement[4].
>> >
>> >>  In our performance testing, we've seen consistent throughput increases
>> >>  across multiple configurations. In some cases, we observed up to 75%
>> >>  increase in events processed through Logstash.
>> >
>> > another benefit to using Logstash is the whole ELK stack is maintained
>> > by
>> > one
>> > community/company. It is well tested and easy to upgrade the whole stack
>> > at
>> > the
>> > same time. Using other tools may force us on certain elasticsearch
>> > release.
>> >
>> > So, I think we have to alternative tools.
>> >
>> > * Fluentd
>> > * Logstash
>> >
>> > IMO, we need to make the decision and at least prepare the migration
>> > solution now.
>> >
>> > [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
>> > [2]
>> >
>> > https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html
>> > [3] http://www.fluentd.org/
>> > [4] https://www.elastic.co/blog/logstash-5-0-0-released
>> >
>> > --
>> > Regards,
>> > Jeffrey Zhang
>> > Blog: http://xcodest.me
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Shake Chen
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Shake Chen

Hi Michal

the snap seem not support log forward now, I can not find any infomation in
google .

On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski 
wrote:

> Hey,
>
> I am also working with Snap community to enable log forwarding with it [1].
> Snap is super lightweight and additional benefit of this solution
> would be that it can also handle monitoring, which was it's initial
> role. One service to handle both would be elegant. I'll keep you
> posted but let's not throw away this idea just yet.
>
> Cheers,
> Michal
>
> [1] https://github.com/intelsdi-x/snap
>
> On 26 November 2016 at 23:55, Jeffrey Zhang 
> wrote:
> > Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
> have a
> > blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
> >
> > For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> > intent
> > to replace the Logstash at all.
> >
> >> Filebeat is based on the Logstash Forwarder source code and replaces
> >> Logstash
> >> Forwarder as the method to use for tailing log files and forwarding them
> >> to
> >> Logstash.
> >
> > Fillebeat is a log transport tool rather than log processing too. I do
> not
> > treat it as an alternative at all.
> >
> > To be honest, I'd like back to Logstash, and Logstash 5.x is released
> with
> > high
> > performance improvement[4].
> >
> >>  In our performance testing, we've seen consistent throughput increases
> >>  across multiple configurations. In some cases, we observed up to 75%
> >>  increase in events processed through Logstash.
> >
> > another benefit to using Logstash is the whole ELK stack is maintained by
> > one
> > community/company. It is well tested and easy to upgrade the whole stack
> at
> > the
> > same time. Using other tools may force us on certain elasticsearch
> release.
> >
> > So, I think we have to alternative tools.
> >
> > * Fluentd
> > * Logstash
> >
> > IMO, we need to make the decision and at least prepare the migration
> > solution now.
> >
> > [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> > [2]
> > https://www.elastic.co/guide/en/beats/filebeat/current/
> migrating-from-logstash-forwarder.html
> > [3] http://www.fluentd.org/
> > [4] https://www.elastic.co/blog/logstash-5-0-0-released
> >
> > --
> > Regards,
> > Jeffrey Zhang
> > Blog: http://xcodest.me
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Shake Chen
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

2016-11-27 Thread Michał Jastrzębski

Hey,

I am also working with Snap community to enable log forwarding with it [1].
Snap is super lightweight and additional benefit of this solution
would be that it can also handle monitoring, which was it's initial
role. One service to handle both would be elegant. I'll keep you
posted but let's not throw away this idea just yet.

Cheers,
Michal

[1] https://github.com/intelsdi-x/snap

On 26 November 2016 at 23:55, Jeffrey Zhang  wrote:
> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a
> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>
> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> intent
> to replace the Logstash at all.
>
>> Filebeat is based on the Logstash Forwarder source code and replaces
>> Logstash
>> Forwarder as the method to use for tailing log files and forwarding them
>> to
>> Logstash.
>
> Fillebeat is a log transport tool rather than log processing too. I do not
> treat it as an alternative at all.
>
> To be honest, I'd like back to Logstash, and Logstash 5.x is released with
> high
> performance improvement[4].
>
>>  In our performance testing, we've seen consistent throughput increases
>>  across multiple configurations. In some cases, we observed up to 75%
>>  increase in events processed through Logstash.
>
> another benefit to using Logstash is the whole ELK stack is maintained by
> one
> community/company. It is well tested and easy to upgrade the whole stack at
> the
> same time. Using other tools may force us on certain elasticsearch release.
>
> So, I think we have to alternative tools.
>
> * Fluentd
> * Logstash
>
> IMO, we need to make the decision and at least prepare the migration
> solution now.
>
> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> [2]
> https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html
> [3] http://www.fluentd.org/
> [4] https://www.elastic.co/blog/logstash-5-0-0-released
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

Re: [openstack-dev] [kolla] the alternative of log processing tool

18 matches

Site Navigation

Mail list logo

Footer information