Re: Kibana Logs Empty

[Eric Wolinetz]

Realized I never replied-all... Re-adding users_list

On Mon, Aug 15, 2016 at 10:58 AM, Eric Wolinetz  wrote:

> Fluentd tries to connect to both "logging-es" | "logging-es-ops" in the
> logging namespace (if you're using the ops deployment) and "kubernetes" in
> the default namespace.  I think in this case it is having trouble
> connecting to the kubernetes service to look up metadata for your
> containers.
>
>
> On Mon, Aug 15, 2016 at 10:54 AM, Frank Liauw  wrote:
>
>> Oh stupid me; I was confused by my own namespaces; was looking at the
>> wrong namespace, thinking that's the one with pods that have an active log
>> stream. The logs are ingested fine, thanks for your assistance! :)
>>
>> On the possible DNS issue of fluentd on one of my nodes, what hostname is
>> fluentd trying to reach when starting up? We did perform some network
>> changes to this particular node to aid public routing, but as far as the
>> routing table is concerned, it should not have made a difference for local
>> traffic.
>>
>> Normal functioning node without public routing changes
>>
>> [root@node1 network-scripts]# route -n
>> Kernel IP routing table
>> Destination Gateway Genmask Flags Metric RefUse
>> Iface
>> 0.0.0.0 10.10.0.5   0.0.0.0 UG10000
>> ens160
>> 10.1.0.00.0.0.0 255.255.0.0 U 0  00
>> tun0
>> 10.10.0.0   0.0.0.0 255.255.0.0 U 10000
>> ens160
>> 172.30.0.0  0.0.0.0 255.255.0.0 U 0  00
>> tun0
>>
>> Malfunctioning node with public routing changes
>>
>> [root@node2 network-scripts]# route -n
>> Kernel IP routing table
>> Destination Gateway Genmask Flags Metric RefUse
>> Iface
>> 0.0.0.0 199.27.105.10.0.0.0 UG10000
>> ens192
>> 10.0.0.010.10.0.5   255.0.0.0   UG10000
>> ens160
>> 10.1.0.00.0.0.0 255.255.0.0 U 0  00
>> tun0
>> 10.10.0.0   0.0.0.0 255.255.0.0 U 10000
>> ens160
>> 172.30.0.0  0.0.0.0 255.255.0.0 U 0  00
>> tun0
>> 199.27.105.00.0.0.0 255.255.255.128 U 10000
>> ens192
>>
>> Frank
>> Systems Engineer
>>
>> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>>
>> Join me on VSee for Free 
>>
>>
>>
>>
>> On Mon, Aug 15, 2016 at 11:23 PM, Eric Wolinetz 
>> wrote:
>>
>>> Correct, the way Fluentd pulls in the logs for your other containers is
>>> the same pipeline used for collecting logs for the below shown Kibana pod.
>>>
>>> Going back to your ES logs, can you verify the date portion of a
>>> microsvc index line?
>>> We can then update time range in the upper-right corner of Kibana to
>>> change from the last hour to something like the last month (something that
>>> would encompass the date for the index).
>>>
>>>
>>> On Mon, Aug 15, 2016 at 10:15 AM, Frank Liauw  wrote:
>>>
 Screencap is as follows:


 The query is as simple as it gets, *. I see my namespaces / projects as
 indexes.

 I see logs for logging project just fine:



 Fluentd is not ingesting the logs for pods in my namespaces. I'm yet to
 pull apart how fluentd does that, though there's no reason why logs for my
 other pods aren't getting indexed whereas kibana logs are if they are both
 ingested by fluentd, assuming that kibana logs use the same pipeline as all
 other pod logs.

 Frank
 Systems Engineer

 VSee: fr...@vsee.com  | Cell: +65 9338 0035

 Join me on VSee for Free 




 On Mon, Aug 15, 2016 at 10:59 PM, Eric Wolinetz 
 wrote:

> Can you either send a screencap of your Kibana console? Or describe
> how you are accessing Kibana and what you are seeing? (e.g. your query
> string, the index you're querying on, the time range for fetched 
> responses)
>
> On Mon, Aug 15, 2016 at 9:55 AM, Frank Liauw  wrote:
>
>> I can see indexes of my namespaces, but nothing going on in actual
>> logs in kibana though.
>>
>> Frank
>> Systems Engineer
>>
>> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>>
>> Join me on VSee for Free 
>>
>>
>>
>>
>> On Mon, Aug 15, 2016 at 10:37 PM, Eric Wolinetz 
>> wrote:
>>
>>> True, we should be able to.  You should be able to see entries in
>>> the master ES node's logs that indices were created.  Based on your log
>>> snippet it should be "One Above All" in this pod: 
>>> logging-es-0w45va6n-2-8m8
>>> 5p
>>>
>>> If we don't see anything 

Re: Kibana Logs Empty

[Luke Meyer]

On Mon, Aug 15, 2016 at 3:54 AM, Frank Liauw  wrote:

> Hi All,
>
> I followed through the instructions on https://docs.openshift.org/
> latest/install_config/aggregate_logging.html and have setup a 3 node ES
> cluster. Fluentd is also deployed on all my nodes.
>
> I am getting kibana logs on the logging project, but all my other projects
> do not have any logs; kibana shows "No results found", with occasional
> errors reading "Discover: An error occurred with your request. Reset your
> inputs and try again."
>

Just to make sure... the default time period in Kibana is to look only 15
minutes in the past - are you sure your projects had logs in the last 15
minutes?
That wouldn't have anything to do with the errors you're seeing though.


>
> Probing the requests made by kibana, some calls to
> /elasticsearch/_msearch?timeout=0_unavailable=true
> =1471245075265 are failing from time to time.
>

That certainly shouldn't be happening. Do you have any more details on how
they're failing? Do they fail to connect, or just get back an error
response code? Not sure if you can tell...


>
> Looking into the ES logs for all 3 cluster pods, I don't see much errors
> to be concerned, with the last error of 2 nodes similar to the following
> which seems to be a known issue with Openshift's setup (
> https://lists.openshift.redhat.com/openshift-archives/users
> /2015-December/msg00078.html) and possibly explains the failed requests
> made by kibana on auto-refresh, but that's a problem for another day:
>
> [2016-08-15 06:53:49,130][INFO ][cluster.service  ] [Gremlin]
> added {[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-
> 2-a0cf0][inet[/10.1.3.3:9300]],}, reason: zen-disco-receive(from master
> [[One Above All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][in
> et[/10.1.2.5:9300]]])
>

This is good, means your cluster is forming...


> [2016-08-15 
> 06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter]
> Error while apply() due to com.floragunn.searchguard.toke
> neval.MalformedConfigurationException: no bypass or execute filters at
> all for action indices:admin/mappings/fields/get
> com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no
> bypass or execute filters at all
>

Unfortunate SearchGuard behavior while the cluster is starting, but nothing
to be concerned about as long as it doesn't continue.


>
> Looking into fluentd logs, one of my nodes is complaining of a
> "getaddrinfo" error:
>
> 2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo:
> Name or service not known"
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
> `initialize'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
> `open'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
> `block in connect'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in
> `timeout'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in
> `connect'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in
> `do_start'
>   2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in
> `start'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-cl
> ient-2.0.0/lib/restclient/request.rb:766:in `transmit'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-cl
> ient-2.0.0/lib/restclient/request.rb:215:in `execute'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-cl
> ient-2.0.0/lib/restclient/request.rb:52:in `execute'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/rest-cl
> ient-2.0.0/lib/restclient/resource.rb:51:in `get'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubecli
> ent-1.1.4/lib/kubeclient/common.rb:328:in `block in api'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubecli
> ent-1.1.4/lib/kubeclient/common.rb:58:in `handle_exception'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubecli
> ent-1.1.4/lib/kubeclient/common.rb:327:in `api'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/kubecli
> ent-1.1.4/lib/kubeclient/common.rb:322:in `api_valid?'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluent-
> plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/
> filter_kubernetes_metadata.rb:167:in `configure'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd
> -0.12.23/lib/fluent/agent.rb:144:in `add_filter'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd
> -0.12.23/lib/fluent/agent.rb:61:in `block in configure'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd
> -0.12.23/lib/fluent/agent.rb:57:in `each'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd
> -0.12.23/lib/fluent/agent.rb:57:in `configure'
>   2016-08-15 03:45:18 -0400 [error]: /opt/app-root/src/gems/fluentd
> 

Kibana Logs Empty

[Frank Liauw]

Hi All,

I followed through the instructions on
https://docs.openshift.org/latest/install_config/aggregate_logging.html and
have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes.

I am getting kibana logs on the logging project, but all my other projects
do not have any logs; kibana shows "No results found", with occasional
errors reading "Discover: An error occurred with your request. Reset your
inputs and try again."

Probing the requests made by kibana, some calls to
/elasticsearch/_msearch?timeout=0_unavailable=true=1471245075265
are
failing from time to time.

Looking into the ES logs for all 3 cluster pods, I don't see much errors to
be concerned, with the last error of 2 nodes similar to the following which
seems to be a known issue with Openshift's setup (
https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html)
and possibly explains the failed requests made by kibana on auto-refresh,
but that's a problem for another day:

[2016-08-15 06:53:49,130][INFO ][cluster.service  ] [Gremlin] added
{[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],},
reason: zen-disco-receive(from master [[One Above
All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300
]]])
[2016-08-15
06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter]
Error while apply() due to
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no
bypass or execute filters at all for action
indices:admin/mappings/fields/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no
bypass or execute filters at all

Looking into fluentd logs, one of my nodes is complaining of a
"getaddrinfo" error:

2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo:
Name or service not known"
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`initialize'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`open'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`block in connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in
`timeout'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in
`connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in
`do_start'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in
`start'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in
`transmit'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in
`execute'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in
`execute'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in
`get'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in
`block in api'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in
`handle_exception'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in
`api'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in
`api_valid?'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in
`add_filter'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in
configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in
`block in configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in
`run_configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in
`run_configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in
`block in start'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in