I'm trying to setup logging on my GKE cluster by following the suggestions from this thread <https://groups.google.com/forum/#!searchin/kubernetes-users/fluentd|sort:relevance/kubernetes-users/Q1nvl8IAbqc/BXOZNBUWAgAJ>, but have hit a wall.
Specifically, I've: 1) Created a new cluster on GKE with google logging disabled 2) Ran `kubectl create -f ...` on each of the following 5 components: # From: https://github.com/kubernetes/kubernetes/tree/dae5ac482861382e18b1e7b2943b1b7f333c6a2a/cluster/addons/fluentd-elasticsearch es-controller.yaml es-service.yaml kibana-controller.yaml kibana-service.yaml # From: https://github.com/kubernetes/kubernetes/blob/714db746118923a3918e28aacb4564e91afdd368/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml fluentd-es.yaml However I'm seeing some odd results. My service/rc/pods for kibana and elastic search are created, then die off very quickly. Here's what it looks like before those components die off: $kubectl get all --namespace=kube-system NAME DESIRED CURRENT AGE elasticsearch-logging-v1 2 2 4s kibana-logging-v1 1 1 3s kube-dns-v17.1 2 2 2h kubernetes-dashboard-v1.1.1 1 1 2h l7-default-backend-v1.0 1 1 2h NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend 10.3.245.50 <nodes> 80/TCP 2h elasticsearch-logging 10.3.242.22 <none> 9200/TCP 5s heapster 10.3.246.252 <none> 80/TCP 2h kibana-logging 10.3.241.94 <nodes> 5601/TCP 3s kube-dns 10.3.240.10 <none> 53/UDP,53/TCP 2h kubernetes-dashboard 10.3.242.135 <none> 80/TCP 2h NAME READY STATUS RESTARTS AGE elasticsearch-logging-v1-lq5gx 1/1 Running 0 4s elasticsearch-logging-v1-p78mn 1/1 Running 0 4s fluentd-elasticsearch 1/1 Running 0 6s heapster-v1.1.0-2096339923-4j1vu 2/2 Running 0 2h kibana-logging-v1-km83q 1/1 Running 0 3s kube-dns-v17.1-aqteq 3/3 Running 0 2h kube-dns-v17.1-qcm9z 3/3 Running 0 2h kube-proxy-gke-pachyderm-log-test-default-pool-8429ab58-1sio 1/1 Running 0 2h kube-proxy-gke-pachyderm-log-test-default-pool-8429ab58-b7tw 1/1 Running 0 2h kube-proxy-gke-pachyderm-log-test-default-pool-8429ab58-eead 1/1 Running 0 2h kubernetes-dashboard-v1.1.1-aajls 1/1 Running 0 2h l7-default-backend-v1.0-filol 1/1 Running 0 2h Normally I'd get the logs using the previous flag, but that doesn't seem to work. However if I get the logs fast enough, I can see a few things, but nothing out of the ordinary. $kubectl logs elasticsearch-logging-v1-lq5gx --namespace=kube-system I0823 21:16:06.814369 5 elasticsearch_logging_discovery.go:42] Kubernetes Elasticsearch logging discovery I0823 21:16:07.829359 5 elasticsearch_logging_discovery.go:75] Found [ "10.0.1.4" "10.0.2.6"] Which looks pretty normal to me. Grabbing the logs from the kibana pod quickly, I see: $kubectl logs kibana-logging-v1-km83q --namespace=kube-system ELASTICSEARCH_URL=http://elasticsearch-logging:9200 {"@timestamp":"2016-08-23T21:16:10.223Z","level":"error","node_env": "production","error":"Request error, retrying -- connect ECONNREFUSED"} {"@timestamp":"2016-08-23T21:16:10.227Z","level":"warn","message":"Unable to revive connection: http://elasticsearch-logging:9200/","node_env": "production"} {"@timestamp":"2016-08-23T21:16:10.227Z","level":"warn","message":"No living connections","node_env":"production"} {"@timestamp":"2016-08-23T21:16:10.229Z","level":"info","message":"Unable to connect to elasticsearch at http://elasticsearch-logging:9200. Retrying in 2.5 seconds.","node_env":"production"} Which makes sense ... considering the elastic search pod dies off very quickly. The fluentd pod reports an error connecting to elastic search, but again, thats not surprising considering it quickly dies off. Clearly, I'm missing some steps in configuring these services to connect to each other. Any advice along these lines is appreciated. I know that I should probably be using a daemon set to spin up the fluentd pod, but I thought I could get these components wired together before worrying about that step. Maybe thats not the case. Either way, I'd appreciate any pointers in setting that up as well. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
