On Wed, Apr 13, 2016 at 3:16 AM, Lorenz Vanthillo < lorenz.vanthi...@outlook.com> wrote:
> I saw on https://github.com/openshift/origin/issues/8358: > > > $ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current > Debugging with pod/debug-logging-fluentd-80xzt, original command: <image > entrypoint> > Waiting for pod to start ... > system_u:system_r:svirt_lxc_net_t:s0:c216,c576 > > Removing debug pod ... > > > Yup. The problem was what I thought: it's being run under the > svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t. If > you don't want to disable SELinux, you'll need to follow the instructions > for creating a new SELinux type that I posted above. > > So I understand what's wrong but I don't see why the workaround (changing > the service account permissions from anyuid to privileged) isn't working > for me + I don't want to create a new selinuxtype. > Sorry about that, we had missed a step. You'll need to delete your daemonset, edit your logging-fluentd-template to add a property to your container spec and recreate your daemonset to let it properly run as privileged to escape the SELinux enforcing. $ oc delete daemonset logging-fluentd $ oc edit template/logging-fluentd-template # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 kind: Template labels: component: fluentd . . . objects: - apiVersion: extensions/v1beta1 kind: DaemonSet . . . spec: selector: matchLabels: component: fluentd provider: openshift template: metadata: labels: component: fluentd provider: openshift name: fluentd-elasticsearch spec: containers: . . . name: fluentd-elasticsearch # insert below here securityContext: privileged: true # insert above here resources: limits: cpu: 100m . . . $ oc process logging-fluentd-template | oc create -f - > ------------------------------ > From: lorenz.vanthi...@outlook.com > To: ewoli...@redhat.com > CC: users@lists.openshift.redhat.com > Subject: RE: Aggregating container logs using Kibana > Date: Wed, 13 Apr 2016 09:30:48 +0200 > > > Fixed the issue with nodeselectormismatching: > So now I have 3 fluentd pods on my 2 normal nodes and my infranode: > But still the same permission issue: > NAME READY STATUS RESTARTS AGE > logging-curator-1-j7mz0 1/1 Running 0 17m > logging-deployer-39qcz 0/1 Completed 0 47m > logging-es-605u5g7g-1-36owl 1/1 Running 0 17m > logging-fluentd-4uqx1 1/1 Running 0 46m > logging-fluentd-dez5r 1/1 Running 0 2m > logging-fluentd-m50nj 1/1 Running 0 46m > logging-kibana-1-wfog2 2/2 Running 0 16m > > ------------------------------ > From: lorenz.vanthi...@outlook.com > To: ewoli...@redhat.com > CC: users@lists.openshift.redhat.com > Subject: RE: Aggregating container logs using Kibana > Date: Wed, 13 Apr 2016 09:21:47 +0200 > > Hi Eric, > > Thanks for your reply and the follow up of this issue. > I've created a new origin 1.1.6 cluster (2 days ago) but still have the > same issue: > My environment is one master (with node) non schedulable, 2 'normal' nodes > and one infra node. > I still got the permission denied (The documentation is up to date so I > even don't had to perform the workaround manually). > - system:serviceaccount:logging:aggregated-logging-fluentd is in scc > privileged by default. > > The logging-deployer-template creates services and 2 pods of fluentd (on > the normal nodes). > The pods appear after performing this command: > > oc label nodes --all logging-infra-fluentd=true > > So my nodes got that label. also the unschedulable node on my master. So > that's normal that it failed but why it fails on my infra-node I don't > know. (I defined in my master-config that projects are by default on the > other 2 nodes, maybe that's why but I don't know it's relevant for my > issue). > I also don't really understand why 'oc process logging-support-tempalte | > oc create -f -' is only be cited at the troubleshooting part. > Still the error: [error]: unexpected error error_class=Errno::EACCES > error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> > > oc get is > NAME DOCKER REPO > TAGS UPDATED > logging-auth-proxy docker.io/openshift/origin-logging-auth-proxy > latest,v0.0.1 4 minutes ago > logging-curator docker.io/openshift/origin-logging-curator > latest 4 minutes ago > logging-elasticsearch docker.io/openshift/origin-logging-elasticsearch > latest 4 minutes ago > logging-fluentd docker.io/openshift/origin-logging-fluentd > latest 4 minutes ago > logging-kibana docker.io/openshift/origin-logging-kibana > latest 4 minutes ago > > oc get svc > NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE > logging-es 172.30.68.xx <none> 9200/TCP 33m > logging-es-cluster None <none> 9300/TCP 33m > logging-es-ops 172.30.18.xx <none> 9200/TCP 33m > logging-es-ops-cluster None <none> 9300/TCP 33m > logging-kibana 172.30.216.xx <none> 443/TCP 33m > logging-kibana-ops 172.30.186.xx <none> 443/TCP 33m > > oc get pods > NAME READY STATUS > RESTARTS AGE > logging-curator-1-j7mz0 1/1 Running > 0 4m > logging-deployer-39qcz 0/1 Completed > 0 34m > logging-es-605u5g7g-1-36owl 1/1 Running > 0 4m > logging-fluentd-4uqx1 1/1 Running > 0 33m > logging-fluentd-ex34j 0/1 NodeSelectorMismatching > 0 33m > logging-fluentd-injz7 0/1 NodeSelectorMismatching > 0 33m > logging-fluentd-m50nj 1/1 Running > 0 33m > logging-kibana-1-wfog2 2/2 Running > 0 4m > > oc get daemonset > NAME DESIRED CURRENT NODE-SELECTOR AGE > logging-fluentd 4 4 logging-infra-fluentd=true 34m > > oc get dc > NAME REVISION REPLICAS TRIGGERED BY > logging-curator 1 1 > config,image(logging-curator:latest) > logging-es-605u5g7g 1 1 > config,image(logging-elasticsearch:latest) > logging-kibana 1 1 > config,image(logging-auth-proxy:latest),image(logging-kibana:latest) > > oc get routes > [centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops) > NAME HOST/PORT PATH SERVICE > TERMINATION LABELS > kibana kibana.test.xxx.eu logging-kibana > passthrough component=support,logging-infra=support,provider=openshift > kibana-ops kibana-ops.example.com logging-kibana-ops > passthrough component=support,logging-infra=support,provider=openshift > > oc get oauthclient > NAME > SECRET > WWW-CHALLENGE REDIRECT URIS > kibana-proxy > j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX FALSE > https://kibana.xxx.eu,https://kibana-ops.example.com > openshift-browser-client > 71724303-b823-4435-8568-bcafxxxx4 > FALSE > https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display > openshift-challenging-client ac7c9942-9a55-4e1e-8e5f-9fxxxxx > TRUE > https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit > openshift-web-console 6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx > FALSE > https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000 > > > > > > > > ------------------------------ > From: ewoli...@redhat.com > Date: Tue, 12 Apr 2016 17:27:06 -0500 > Subject: Re: Aggregating container logs using Kibana > To: lorenz.vanthi...@outlook.com > CC: lme...@redhat.com; users@lists.openshift.redhat.com > > > > On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo < > lorenz.vanthi...@outlook.com> wrote: > > This are all the steps I'm performing: > > oc new-project logging > > $ oc secrets new logging-deployer nothing=/dev/null > > $ oc process logging-deployer-account-template -n openshift \ > | oc create -f - > > $ oc policy add-role-to-user edit --serviceaccount logging-deployer > $ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer > $ oadm policy add-cluster-role-to-user oauth-editor \ > system:serviceaccount:logging:logging-deployer > > $ oadm policy add-scc-to-user \ > privileged system:serviceaccount:logging:aggregated-logging-fluentd > > $ oadm policy add-cluster-role-to-user cluster-reader \ > system:serviceaccount:logging:aggregated-logging-fluentd > > Than I execute the deployer template: > > $ oc process logging-deployer-template -n openshift \ > -v > KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 > \ > | oc create -f - > > This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on > master machine) and some empty services (the logs of the pods are telling me > the permission error) > When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all > fine. > > $ oc label nodes --all logging-infra-fluentd=true > > I've edited */master/master-config.yaml* + restart > $ oc scale dc/logging-kibana --replicas=2 > > > $ oc delete oauthclient/kibana-proxy > $ oc process logging-support-template | oc create -f - > > The last step creates also some pods. It's a bit weird for me that this step > is only mentioned for troubleshooting or is > it an issue that I don't have those pods after executing the > deployer-template? > > The template 'logging-support-template' creates your ImageStreams (along > with your routes and oauthclient) so it shouldn't be creating your pods. > There may have been a delay in scheduling your pods initially or the image > stream tags could have been in the processes of being fetched. > > What does the following output? > oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging > > And do you still see the same permission denied errors in the Fluentd logs? > > > > ------------------------------ > From: lorenz.vanthi...@outlook.com > To: lme...@redhat.com > CC: users@lists.openshift.redhat.com > Subject: RE: Aggregating container logs using Kibana > Date: Tue, 5 Apr 2016 18:00:02 +0200 > > > I still have the same issue: > > I've deleted it from scc hostmount-anyuid and added it on scc privileged. > I've deleted all fluentd pods but still the same issue. Even after > recreating the project. > > ------------------------------ > From: lme...@redhat.com > Date: Tue, 5 Apr 2016 10:29:04 -0400 > Subject: Re: Aggregating container logs using Kibana > To: lorenz.vanthi...@outlook.com > CC: users@lists.openshift.redhat.com > > > > On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <lme...@redhat.com> wrote: > > > 2016-04-05 10:55:13 +0000 [error]: unexpected error > error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - > /var/log/es-containers.log.pos> > > > This looks like > https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps > fluentd from reading any logs on the node. > > You should be able to resolve this by adding the fluentd service account > to the privileged SCC, then having fluentd restart everywhere. > > oadm policy add-scc-to-user privileged > system:serviceaccount:logging:aggregated-logging-fluentd > > > Oh; probably need to also remove them from the hostmount-anyuid SCC. > > > _______________________________________________ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
