Re: [Logging] What component forward log entries to fluentd input service?

2017-07-11 Thread Stéphane Klein 
2017-07-11 15:00 GMT+02:00 Alex Wauck :

> Last I checked (OpenShift Origin 1.2), fluentd was just slurping up the
> log files produced by Docker.  It can do that because the pods it runs in
> have access to the host filesystem.
>
> On Tue, Jul 11, 2017 at 6:12 AM, Stéphane Klein <
> cont...@stephane-klein.info> wrote:
>
>> Hi,
>>
>> I see here https://github.com/openshift/origin-aggregated-logging/
>> blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2
>> that fluentd logging system use secure_forward input system.
>>
>> My question: what component forward log entries to fluentd input service ?
>>
>>
Ok it's here:

bash-4.2# cat configs.d/dynamic/input-syslog-default-syslog.conf

  @type systemd
  @label @INGRESS
  path "/var/log/journal"
  pos_file /var/log/journal.pos
  tag journal


Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: [Logging] What component forward log entries to fluentd input service?

2017-07-11 Thread Richard Megginson 
Please see 
https://github.com/openshift/origin-aggregated-logging/blob/master/docs/mux-logging-service.md

- Original Message -
> Hi,
> 
> I see here
> https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2
> 
> that fluentd logging system use secure_forward input system.
> 
> My question: what component forward log entries to fluentd input service ?
> 
> Best regards,
> Stéphane
> --
> Stéphane Klein 
> blog: http://stephane-klein.info
> cv : http://cv.stephane-klein.info
> Twitter: http://twitter.com/klein_stephane
> 

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: [Logging] What component forward log entries to fluentd input service?

2017-07-11 Thread Peter Portante 
On Tue, Jul 11, 2017 at 9:00 AM, Alex Wauck  wrote:
> Last I checked (OpenShift Origin 1.2), fluentd was just slurping up the log
> files produced by Docker.  It can do that because the pods it runs in have
> access to the host filesystem.
>
> On Tue, Jul 11, 2017 at 6:12 AM, Stéphane Klein
>  wrote:
>>
>> Hi,
>>
>> I see here
>> https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2
>> that fluentd logging system use secure_forward input system.
>>
>> My question: what component forward log entries to fluentd input service ?

The "mux" service is a concentrator of sorts.

Without the mux service, each fluentd pod runs on a host in an
OpenShift cluster collecting logs and sending them to Elasticsearch
directly.  The collectors also have the responsibility of enhancing
the logs collected with the metadata that describes which
pod/container they came from.  This requires connections to the API
server to get that information.

So in a large cluster, 200+ nodes, maybe less, maybe more, the API
servers are overwhelmed by requests from all the fluentd pods.

With the mux service, all the fluentd collections pods only talk to
the mux service and DO NOT talk to the API server; they simply send
the logs they collect to the mux fluentd instance.

The mux fluentd instance in turns talks to the API service to enrich
the logs with the pod/container metadata and then send along to
Elasticsearch.

This scales much better.

-peter


>>
>> Best regards,
>> Stéphane
>> --
>> Stéphane Klein 
>> blog: http://stephane-klein.info
>> cv : http://cv.stephane-klein.info
>> Twitter: http://twitter.com/klein_stephane
>>
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>
>
>
> --
>
> Alex Wauck // Senior DevOps Engineer
>
> E X O S I T E
> www.exosite.com
>
> Making Machines More Human.
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: [Logging] What component forward log entries to fluentd input service?

2017-07-11 Thread Alex Wauck 
Last I checked (OpenShift Origin 1.2), fluentd was just slurping up the log
files produced by Docker.  It can do that because the pods it runs in have
access to the host filesystem.

On Tue, Jul 11, 2017 at 6:12 AM, Stéphane Klein  wrote:

> Hi,
>
> I see here https://github.com/openshift/origin-aggregated-
> logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2
> that fluentd logging system use secure_forward input system.
>
> My question: what component forward log entries to fluentd input service ?
>
> Best regards,
> Stéphane
> --
> Stéphane Klein 
> blog: http://stephane-klein.info
> cv : http://cv.stephane-klein.info
> Twitter: http://twitter.com/klein_stephane
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 

Alex Wauck // Senior DevOps Engineer

*E X O S I T E*
*www.exosite.com *

Making Machines More Human.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

[Logging] What component forward log entries to fluentd input service?

2017-07-11 Thread Stéphane Klein 
Hi,

I see here
https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L2

that fluentd logging system use secure_forward input system.

My question: what component forward log entries to fluentd input service ?

Best regards,
Stéphane
-- 
Stéphane Klein 
blog: http://stephane-klein.info
cv : http://cv.stephane-klein.info
Twitter: http://twitter.com/klein_stephane
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users