Realized I never replied-all... Re-adding users_list On Mon, Aug 15, 2016 at 10:58 AM, Eric Wolinetz <ewoli...@redhat.com> wrote:
> Fluentd tries to connect to both "logging-es" | "logging-es-ops" in the > logging namespace (if you're using the ops deployment) and "kubernetes" in > the default namespace. I think in this case it is having trouble > connecting to the kubernetes service to look up metadata for your > containers. > > > On Mon, Aug 15, 2016 at 10:54 AM, Frank Liauw <fr...@vsee.com> wrote: > >> Oh stupid me; I was confused by my own namespaces; was looking at the >> wrong namespace, thinking that's the one with pods that have an active log >> stream. The logs are ingested fine, thanks for your assistance! :) >> >> On the possible DNS issue of fluentd on one of my nodes, what hostname is >> fluentd trying to reach when starting up? We did perform some network >> changes to this particular node to aid public routing, but as far as the >> routing table is concerned, it should not have made a difference for local >> traffic. >> >> Normal functioning node without public routing changes >> >> [root@node1 network-scripts]# route -n >> Kernel IP routing table >> Destination Gateway Genmask Flags Metric Ref Use >> Iface >> 0.0.0.0 10.10.0.5 0.0.0.0 UG 100 0 0 >> ens160 >> 10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 >> tun0 >> 10.10.0.0 0.0.0.0 255.255.0.0 U 100 0 0 >> ens160 >> 172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 >> tun0 >> >> Malfunctioning node with public routing changes >> >> [root@node2 network-scripts]# route -n >> Kernel IP routing table >> Destination Gateway Genmask Flags Metric Ref Use >> Iface >> 0.0.0.0 199.27.105.1 0.0.0.0 UG 100 0 0 >> ens192 >> 10.0.0.0 10.10.0.5 255.0.0.0 UG 100 0 0 >> ens160 >> 10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 >> tun0 >> 10.10.0.0 0.0.0.0 255.255.0.0 U 100 0 0 >> ens160 >> 172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 >> tun0 >> 199.27.105.0 0.0.0.0 255.255.255.128 U 100 0 0 >> ens192 >> >> Frank >> Systems Engineer >> >> VSee: fr...@vsee.com <http://vsee.com/u/tmd4RB> | Cell: +65 9338 0035 >> >> Join me on VSee for Free <http://vsee.com/u/tmd4RB> >> >> >> >> >> On Mon, Aug 15, 2016 at 11:23 PM, Eric Wolinetz <ewoli...@redhat.com> >> wrote: >> >>> Correct, the way Fluentd pulls in the logs for your other containers is >>> the same pipeline used for collecting logs for the below shown Kibana pod. >>> >>> Going back to your ES logs, can you verify the date portion of a >>> microsvc index line? >>> We can then update time range in the upper-right corner of Kibana to >>> change from the last hour to something like the last month (something that >>> would encompass the date for the index). >>> >>> >>> On Mon, Aug 15, 2016 at 10:15 AM, Frank Liauw <fr...@vsee.com> wrote: >>> >>>> Screencap is as follows: >>>> >>>> >>>> The query is as simple as it gets, *. I see my namespaces / projects as >>>> indexes. >>>> >>>> I see logs for logging project just fine: >>>> >>>> >>>> >>>> Fluentd is not ingesting the logs for pods in my namespaces. I'm yet to >>>> pull apart how fluentd does that, though there's no reason why logs for my >>>> other pods aren't getting indexed whereas kibana logs are if they are both >>>> ingested by fluentd, assuming that kibana logs use the same pipeline as all >>>> other pod logs. >>>> >>>> Frank >>>> Systems Engineer >>>> >>>> VSee: fr...@vsee.com <http://vsee.com/u/tmd4RB> | Cell: +65 9338 0035 >>>> >>>> Join me on VSee for Free <http://vsee.com/u/tmd4RB> >>>> >>>> >>>> >>>> >>>> On Mon, Aug 15, 2016 at 10:59 PM, Eric Wolinetz <ewoli...@redhat.com> >>>> wrote: >>>> >>>>> Can you either send a screencap of your Kibana console? Or describe >>>>> how you are accessing Kibana and what you are seeing? (e.g. your query >>>>> string, the index you're querying on, the time range for fetched >>>>> responses) >>>>> >>>>> On Mon, Aug 15, 2016 at 9:55 AM, Frank Liauw <fr...@vsee.com> wrote: >>>>> >>>>>> I can see indexes of my namespaces, but nothing going on in actual >>>>>> logs in kibana though. >>>>>> >>>>>> Frank >>>>>> Systems Engineer >>>>>> >>>>>> VSee: fr...@vsee.com <http://vsee.com/u/tmd4RB> | Cell: +65 9338 0035 >>>>>> >>>>>> Join me on VSee for Free <http://vsee.com/u/tmd4RB> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Aug 15, 2016 at 10:37 PM, Eric Wolinetz <ewoli...@redhat.com> >>>>>> wrote: >>>>>> >>>>>>> True, we should be able to. You should be able to see entries in >>>>>>> the master ES node's logs that indices were created. Based on your log >>>>>>> snippet it should be "One Above All" in this pod: >>>>>>> logging-es-0w45va6n-2-8m8 >>>>>>> 5p >>>>>>> >>>>>>> If we don't see anything like "[logging.iasdf9123j-asdf.2016.02.19] >>>>>>> creating index" then there might be something else going on... Do you >>>>>>> see >>>>>>> any entries like that for your projects? >>>>>>> >>>>>>> On Mon, Aug 15, 2016 at 9:05 AM, Frank Liauw <fr...@vsee.com> wrote: >>>>>>> >>>>>>>> I reckoned the same, and already tried doing that, but still got >>>>>>>> the same message. That node have the same configuration as all my other >>>>>>>> nodes, so I don't see why that's the case. >>>>>>>> >>>>>>>> Even if it is having DNS issues, I should still be seeing logs from >>>>>>>> all other nodes that didn't report that error? >>>>>>>> >>>>>>>> Frank >>>>>>>> Systems Engineer >>>>>>>> >>>>>>>> VSee: fr...@vsee.com <http://vsee.com/u/tmd4RB> | Cell: +65 9338 >>>>>>>> 0035 >>>>>>>> >>>>>>>> Join me on VSee for Free <http://vsee.com/u/tmd4RB> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 15, 2016 at 9:49 PM, Eric Wolinetz <ewoli...@redhat.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> It looks like that one Fluentd pod is having DNS issues making a >>>>>>>>> connection to the Kubernetes service. Can you try deleting that pod >>>>>>>>> to >>>>>>>>> have a new one redeployed and see if that is in a better state? >>>>>>>>> >>>>>>>>> On Mon, Aug 15, 2016 at 2:54 AM, Frank Liauw <fr...@vsee.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I followed through the instructions on >>>>>>>>>> https://docs.openshift.org/latest/install_config/aggregat >>>>>>>>>> e_logging.html and have setup a 3 node ES cluster. Fluentd is >>>>>>>>>> also deployed on all my nodes. >>>>>>>>>> >>>>>>>>>> I am getting kibana logs on the logging project, but all my other >>>>>>>>>> projects do not have any logs; kibana shows "No results found", with >>>>>>>>>> occasional errors reading "Discover: An error occurred with your >>>>>>>>>> request. >>>>>>>>>> Reset your inputs and try again." >>>>>>>>>> >>>>>>>>>> Probing the requests made by kibana, some calls to >>>>>>>>>> /elasticsearch/_msearch?timeout=0&ignore_unavailable=true >>>>>>>>>> &preference=1471245075265 are failing from time to time. >>>>>>>>>> >>>>>>>>>> Looking into the ES logs for all 3 cluster pods, I don't see much >>>>>>>>>> errors to be concerned, with the last error of 2 nodes similar to the >>>>>>>>>> following which seems to be a known issue with Openshift's setup ( >>>>>>>>>> https://lists.openshift.redhat.com/openshift-archives/users >>>>>>>>>> /2015-December/msg00078.html) and possibly explains the failed >>>>>>>>>> requests made by kibana on auto-refresh, but that's a problem for >>>>>>>>>> another >>>>>>>>>> day: >>>>>>>>>> >>>>>>>>>> [2016-08-15 06:53:49,130][INFO ][cluster.service ] >>>>>>>>>> [Gremlin] added {[Quicksilver][t2l6Oz8uT-WS8Fa >>>>>>>>>> 7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],}, >>>>>>>>>> reason: zen-disco-receive(from master [[One Above >>>>>>>>>> All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][in >>>>>>>>>> et[/10.1.2.5:9300]]]) >>>>>>>>>> [2016-08-15 06:59:27,727][ERROR][com.flora >>>>>>>>>> gunn.searchguard.filter.SearchGuardActionFilter] Error while >>>>>>>>>> apply() due to com.floragunn.searchguard.toke >>>>>>>>>> neval.MalformedConfigurationException: no bypass or execute >>>>>>>>>> filters at all for action indices:admin/mappings/fields/get >>>>>>>>>> com.floragunn.searchguard.tokeneval.MalformedConfigurationException: >>>>>>>>>> no bypass or execute filters at all >>>>>>>>>> >>>>>>>>>> Looking into fluentd logs, one of my nodes is complaining of a >>>>>>>>>> "getaddrinfo" error: >>>>>>>>>> >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: unexpected error >>>>>>>>>> error="getaddrinfo: Name or service not known" >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:878:in `initialize' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:878:in `open' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:878:in `block in connect' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/timeout.rb:52:in `timeout' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:877:in `connect' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:862:in `do_start' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/ruby/net/http.rb:851:in `start' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in >>>>>>>>>> `transmit' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in >>>>>>>>>> `execute' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in >>>>>>>>>> `execute' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in >>>>>>>>>> `get' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in >>>>>>>>>> `block in api' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in >>>>>>>>>> `handle_exception' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in >>>>>>>>>> `api' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in >>>>>>>>>> `api_valid?' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_fil >>>>>>>>>> ter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in >>>>>>>>>> `configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in >>>>>>>>>> `add_filter' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in >>>>>>>>>> `block in configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in >>>>>>>>>> `each' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in >>>>>>>>>> `configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in >>>>>>>>>> `block in configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in >>>>>>>>>> `each' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in >>>>>>>>>> `configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in >>>>>>>>>> `configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in >>>>>>>>>> `run_configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in >>>>>>>>>> `run_configure' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in >>>>>>>>>> `block in start' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in >>>>>>>>>> `call' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in >>>>>>>>>> `main_process' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in >>>>>>>>>> `block in supervise' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in >>>>>>>>>> `fork' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in >>>>>>>>>> `supervise' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in >>>>>>>>>> `start' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in >>>>>>>>>> `<top (required)>' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in >>>>>>>>>> `require' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in >>>>>>>>>> `require' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/gems/fluentd-0.12.23/bin/fluentd:5:in `<top >>>>>>>>>> (required)>' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/bin/fluentd:23:in `load' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: >>>>>>>>>> /opt/app-root/src/bin/fluentd:23:in `<main>' >>>>>>>>>> 2016-08-15 03:45:18 -0400 [error]: fluentd main process died >>>>>>>>>> unexpectedly. restarting. >>>>>>>>>> >>>>>>>>>> Other fluentd pods are fine: >>>>>>>>>> >>>>>>>>>> 2016-08-15 02:55:58 -0400 [info]: reading config file >>>>>>>>>> path="/etc/fluent/fluent.conf" >>>>>>>>>> >>>>>>>>>> However, I'm not getting any logs on pods running on nodes that >>>>>>>>>> have fluentd deployed properly. >>>>>>>>>> >>>>>>>>>> Any thoughts will be much appreciated :) >>>>>>>>>> >>>>>>>>>> Frank >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> users@lists.openshift.redhat.com >>>>>>>>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
