Kubernetes plugin cannot start POD´s due to PVC creation error
Hi, I´m running Jenkins Version 2.190.1 in an openShift 3.9 Cluster, Kubernetes plugin is at version 1.19.3 Since one of the last updates, I sometimes run into: [id=1597]WARNINGo.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j, template=PodTemplate{inheritFrom='', name='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x', namespace='', label='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace. DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{name='main', image= 'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-oc:latest' , alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net , getKey()=LOCAL_URL], KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus , getKey()=NEXUS_URL], KeyValueEnvVar [getValue()=default, getKey()=clusterName], KeyValueEnvVar [getValue()=rspsales-ci, getKey()=project], KeyValueEnvVar [getValue()=BuildConfig.yml, getKey()=buildConfigFile]]}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8 , org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=[ apiVersion: v1 kind: Pod metadata: labels: tier: ci cinextProject: null app: jenkins-slave spec: containers: - name: jnlp image: 'jenkins/jnlp-slave:alpine' args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)'] resources: limits: cpu: '200m' memory: '256Mi' requests: cpu: '200m' memory: '128Mi' env: - name: JAVA_OPTS value: '-Xmx128m' ]} io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.221.128.1/api/v1/namespaces/rspsales-ci/persistentvolumeclaims . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims "pvc-b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User "system:serviceaccount:rspsales-ci:jenkins" cannot update pods/finalizers in project "rspsales-ci", . at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure (OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport. assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse (OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse (OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate( OperationSupport.java:241) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate( BaseOperation.java:813) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create( BaseOperation.java:328) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create( BaseOperation.java:324) at org.csanchez.jenkins.plugins.kubernetes.volumes.workspace. DynamicPVCWorkspaceVolume.createVolume(DynamicPVCWorkspaceVolume.java:94) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch( KubernetesLauncher.java:130) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call( ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call( ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. ja)va:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:624) at java.lang.Thread.run(Thread.java:748) I guess it´s related to the Dynamic PVC´s (see JENKINS-47591) introduced in 1.19.2 - but how can this be resolved ? The strange thing about it is that after restarting Jenkins the POD launching works several times - and than suddenly starts to fail with the above message. I´m running Jenkins with a dedicated service-account:jenkins at openShift, having either "edit" or now for testing "admin" role. Thanx for any ideas, Torsten -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cdf14f70-981f-49e9-9b6a-16621910626c%40googlegroups.co
Re: Kubernetes plugin cannot start POD´s due to PVC creation error
I´ve enabled a "kubernetes" Logger with Level.FINEST and got this output: Combining pod templates, parent: PodTemplate{inheritFrom='', name='default', namespace='', label='', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace. DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{name= 'main', image= 'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/jenkins-slave/base:latest' , workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins. kubernetes.ContainerLivenessProbe@16be1b19}]} Oct 17, 2019 1:55:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils Combining pod templates, template: PodTemplate{, name= '9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', label= '9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeUsageMode=EXCLUSIVE, containers =[ContainerTemplate{name='main', image= 'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest' , alwaysPullImage=true, command='/bin/sh -c', args='cat', ttyEnabled=true, envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net, getKey()=LOCAL_URL], KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus, getKey()=NEXUS_URL]]}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]} Oct 17, 2019 1:55:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes. PodTemplateUtils Pod templates combined: PodTemplate{inheritFrom='', name= '9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', namespace='', label= '9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeSelector='', nodeUsageMode= EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes. workspace.DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{ name='main', image= 'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest' , alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net, getKey()=LOCAL_URL], KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus, getKey()=NEXUS_URL]]}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls= I´m wondering why the parent PodTemplate has workspaceVolume=DynamicPVCWorkspaceVolume? Is this the new default? I´ll try to enable a custom Workspace (EmptyDir) in Kubernetes plugin configuration - which should prevent from creation of DynamicPVC´s causing the error. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8829441d-c93e-4546-bee2-b9d7358c7ebf%40googlegroups.com.