On Wed, Apr 13, 2016 at 3:16 AM, Lorenz Vanthillo <
lorenz.vanthi...@outlook.com> wrote:

> I saw on https://github.com/openshift/origin/issues/8358:
>
>
> $ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current
> Debugging with pod/debug-logging-fluentd-80xzt, original command: <image 
> entrypoint>
> Waiting for pod to start ...
> system_u:system_r:svirt_lxc_net_t:s0:c216,c576
>
> Removing debug pod ...
>
>
> Yup. The problem was what I thought: it's being run under the
> svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t. If
> you don't want to disable SELinux, you'll need to follow the instructions
> for creating a new SELinux type that I posted above.
>
> So I understand what's wrong but I don't see why the workaround (changing
> the service account permissions from anyuid to privileged) isn't working
> for me + I don't want to create a new selinuxtype.
>

Sorry about that, we had missed a step.  You'll need to delete your
daemonset, edit your logging-fluentd-template to add a property to your
container spec and recreate your daemonset to let it properly run as
privileged to escape the SELinux enforcing.

$ oc delete daemonset logging-fluentd

$ oc edit template/logging-fluentd-template


# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving
this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Template
labels:
  component: fluentd
. . .
objects:
- apiVersion: extensions/v1beta1
  kind: DaemonSet
. . .
spec:
    selector:
      matchLabels:
        component: fluentd
        provider: openshift
    template:
      metadata:
        labels:
          component: fluentd
          provider: openshift
        name: fluentd-elasticsearch
      spec:
        containers:
. . .
          name: fluentd-elasticsearch

# insert below here
          securityContext:
            privileged: true
# insert above here

          resources:
            limits:
              cpu: 100m
. . .

$ oc process logging-fluentd-template | oc create -f -


> ------------------------------
> From: lorenz.vanthi...@outlook.com
> To: ewoli...@redhat.com
> CC: users@lists.openshift.redhat.com
> Subject: RE: Aggregating container logs using Kibana
> Date: Wed, 13 Apr 2016 09:30:48 +0200
>
>
> Fixed the issue with nodeselectormismatching:
> So now I have 3 fluentd pods on my 2 normal nodes and my infranode:
> But still the same permission issue:
> NAME                          READY     STATUS      RESTARTS   AGE
> logging-curator-1-j7mz0       1/1       Running     0          17m
> logging-deployer-39qcz        0/1       Completed   0          47m
> logging-es-605u5g7g-1-36owl   1/1       Running     0          17m
> logging-fluentd-4uqx1         1/1       Running     0          46m
> logging-fluentd-dez5r         1/1       Running     0          2m
> logging-fluentd-m50nj         1/1       Running     0          46m
> logging-kibana-1-wfog2        2/2       Running     0          16m
>
> ------------------------------
> From: lorenz.vanthi...@outlook.com
> To: ewoli...@redhat.com
> CC: users@lists.openshift.redhat.com
> Subject: RE: Aggregating container logs using Kibana
> Date: Wed, 13 Apr 2016 09:21:47 +0200
>
> Hi Eric,
>
> Thanks for your reply and the follow up of this issue.
> I've created a new origin 1.1.6 cluster (2 days ago) but still have the
> same issue:
> My environment is one master (with node) non schedulable, 2 'normal' nodes
> and one infra node.
> I still got the permission denied (The documentation is up to date so I
> even don't had to perform the workaround manually).
> - system:serviceaccount:logging:aggregated-logging-fluentd is in scc
> privileged by default.
>
> The logging-deployer-template creates services and 2 pods of fluentd (on
> the normal nodes).
> The pods appear after performing this command:
>
> oc label nodes --all logging-infra-fluentd=true
>
> So my nodes got that label. also the unschedulable node on my master. So
> that's normal that it failed but why it fails on my infra-node I don't
> know. (I defined in my master-config that projects are by default on the
> other 2 nodes, maybe that's why but I don't know it's relevant for my
> issue).
> I also don't really understand why 'oc process logging-support-tempalte |
> oc create -f -' is only be cited at the troubleshooting part.
> Still the error: [error]: unexpected error error_class=Errno::EACCES
> error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>
>
> oc get is
> NAME                    DOCKER REPO
> TAGS            UPDATED
> logging-auth-proxy      docker.io/openshift/origin-logging-auth-proxy
> latest,v0.0.1   4 minutes ago
> logging-curator         docker.io/openshift/origin-logging-curator
> latest          4 minutes ago
> logging-elasticsearch   docker.io/openshift/origin-logging-elasticsearch
> latest          4 minutes ago
> logging-fluentd         docker.io/openshift/origin-logging-fluentd
> latest          4 minutes ago
> logging-kibana          docker.io/openshift/origin-logging-kibana
> latest          4 minutes ago
>
> oc get svc
> NAME                     CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
> logging-es               172.30.68.xx     <none>        9200/TCP   33m
> logging-es-cluster       None             <none>        9300/TCP   33m
> logging-es-ops           172.30.18.xx    <none>        9200/TCP   33m
> logging-es-ops-cluster   None             <none>        9300/TCP   33m
> logging-kibana           172.30.216.xx   <none>        443/TCP    33m
> logging-kibana-ops       172.30.186.xx   <none>        443/TCP    33m
>
> oc get pods
> NAME                          READY     STATUS
> RESTARTS   AGE
> logging-curator-1-j7mz0       1/1       Running
> 0          4m
> logging-deployer-39qcz        0/1       Completed
> 0          34m
> logging-es-605u5g7g-1-36owl   1/1       Running
> 0          4m
> logging-fluentd-4uqx1         1/1       Running
> 0          33m
> logging-fluentd-ex34j         0/1       NodeSelectorMismatching
> 0          33m
> logging-fluentd-injz7         0/1       NodeSelectorMismatching
> 0          33m
> logging-fluentd-m50nj         1/1       Running
> 0          33m
> logging-kibana-1-wfog2        2/2       Running
> 0          4m
>
> oc get daemonset
> NAME              DESIRED   CURRENT   NODE-SELECTOR                AGE
> logging-fluentd   4         4         logging-infra-fluentd=true   34m
>
> oc get dc
> NAME                  REVISION   REPLICAS   TRIGGERED BY
> logging-curator       1          1
> config,image(logging-curator:latest)
> logging-es-605u5g7g   1          1
> config,image(logging-elasticsearch:latest)
> logging-kibana        1          1
> config,image(logging-auth-proxy:latest),image(logging-kibana:latest)
>
> oc get routes
> [centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops)
> NAME         HOST/PORT                PATH      SERVICE
> TERMINATION   LABELS
> kibana       kibana.test.xxx.eu               logging-kibana
> passthrough   component=support,logging-infra=support,provider=openshift
> kibana-ops   kibana-ops.example.com             logging-kibana-ops
> passthrough   component=support,logging-infra=support,provider=openshift
>
> oc get oauthclient
> NAME
> SECRET
> WWW-CHALLENGE   REDIRECT URIS
> kibana-proxy
> j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX   FALSE
> https://kibana.xxx.eu,https://kibana-ops.example.com
> openshift-browser-client
> 71724303-b823-4435-8568-bcafxxxx4
> FALSE
> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display
> openshift-challenging-client   ac7c9942-9a55-4e1e-8e5f-9fxxxxx
>                              TRUE
> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit
> openshift-web-console          6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx
>                            FALSE
> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000
>
>
>
>
>
>
>
> ------------------------------
> From: ewoli...@redhat.com
> Date: Tue, 12 Apr 2016 17:27:06 -0500
> Subject: Re: Aggregating container logs using Kibana
> To: lorenz.vanthi...@outlook.com
> CC: lme...@redhat.com; users@lists.openshift.redhat.com
>
>
>
> On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo <
> lorenz.vanthi...@outlook.com> wrote:
>
> This are all the steps I'm performing:
>
> oc new-project logging
>
> $ oc secrets new logging-deployer nothing=/dev/null
>
> $ oc process logging-deployer-account-template -n openshift \
>      | oc create -f -
>
> $ oc policy add-role-to-user edit --serviceaccount logging-deployer
> $ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer
> $ oadm policy add-cluster-role-to-user oauth-editor \
>        system:serviceaccount:logging:logging-deployer
>
> $ oadm policy add-scc-to-user  \
>     privileged system:serviceaccount:logging:aggregated-logging-fluentd
>
> $ oadm policy add-cluster-role-to-user cluster-reader \
>     system:serviceaccount:logging:aggregated-logging-fluentd
>
> Than I execute the deployer template:
>
> $ oc process logging-deployer-template -n openshift \
>            -v 
> KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443
>  \
>            | oc create -f -
>
> This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on 
> master machine) and some empty services (the logs of the pods are telling me 
> the permission error)
> When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all 
> fine.
>
> $ oc label nodes --all logging-infra-fluentd=true
>
> I've edited */master/master-config.yaml* + restart
> $ oc scale dc/logging-kibana --replicas=2
>
>
> $ oc delete oauthclient/kibana-proxy
> $ oc process logging-support-template | oc create -f -
>
> The last step creates also some pods. It's a bit weird for me that this step 
> is only mentioned for troubleshooting or is
> it an issue that I don't have those pods after executing the 
> deployer-template?
>
> The template 'logging-support-template' creates your ImageStreams (along
> with your routes and oauthclient) so it shouldn't be creating your pods.
> There may have been a delay in scheduling your pods initially or the image
> stream tags could have been in the processes of being fetched.
>
> What does the following output?
> oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging
>
> And do you still see the same permission denied errors in the Fluentd logs?
>
>
>
> ------------------------------
> From: lorenz.vanthi...@outlook.com
> To: lme...@redhat.com
> CC: users@lists.openshift.redhat.com
> Subject: RE: Aggregating container logs using Kibana
> Date: Tue, 5 Apr 2016 18:00:02 +0200
>
>
> I still have the same issue:
>
> I've deleted it from scc hostmount-anyuid and added it on scc privileged.
> I've deleted all fluentd pods but still the same issue. Even after
> recreating the project.
>
> ------------------------------
> From: lme...@redhat.com
> Date: Tue, 5 Apr 2016 10:29:04 -0400
> Subject: Re: Aggregating container logs using Kibana
> To: lorenz.vanthi...@outlook.com
> CC: users@lists.openshift.redhat.com
>
>
>
> On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <lme...@redhat.com> wrote:
>
>
> 2016-04-05 10:55:13 +0000 [error]: unexpected error
> error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied -
> /var/log/es-containers.log.pos>
>
>
> This looks like
> https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps
> fluentd from reading any logs on the node.
>
> You should be able to resolve this by adding the fluentd service account
> to the privileged SCC, then having fluentd restart everywhere.
>
>  oadm policy add-scc-to-user privileged 
> system:serviceaccount:logging:aggregated-logging-fluentd
>
>
> Oh; probably need to also remove them from the  hostmount-anyuid SCC.
>
>
> _______________________________________________
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
_______________________________________________
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Reply via email to