Kubernetes plugin cannot start POD´s due to PVC creation error
Hi,
I´m running Jenkins Version 2.190.1 in an openShift 3.9 Cluster, Kubernetes
plugin is at version 1.19.3
Since one of the last updates, I sometimes run into:
[id=1597]WARNINGo.c.j.p.k.KubernetesLauncher#launch: Error in
provisioning; agent=KubernetesSlave name:
b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j,
template=PodTemplate{inheritFrom='',
name='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x',
namespace='',
label='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa',
nodeSelector='',
nodeUsageMode=EXCLUSIVE,
workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.
DynamicPVCWorkspaceVolume@79ebc880,
containers=[ContainerTemplate{name='main', image=
'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-oc:latest'
,
alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh
-c', args='cat', ttyEnabled=true, resourceRequestCpu='',
resourceRequestMemory='',
resourceLimitCpu='',
resourceLimitMemory='',
envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net
, getKey()=LOCAL_URL],
KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus
, getKey()=NEXUS_URL],
KeyValueEnvVar [getValue()=default, getKey()=clusterName],
KeyValueEnvVar [getValue()=rspsales-ci, getKey()=project],
KeyValueEnvVar [getValue()=BuildConfig.yml, getKey()=buildConfigFile]]}],
annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8
, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=[
apiVersion: v1
kind: Pod
metadata:
labels:
tier: ci
cinextProject: null
app: jenkins-slave
spec:
containers:
- name: jnlp
image: 'jenkins/jnlp-slave:alpine'
args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
resources:
limits:
cpu: '200m'
memory: '256Mi'
requests:
cpu: '200m'
memory: '128Mi'
env:
- name: JAVA_OPTS
value: '-Xmx128m'
]}
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
POST at:
https://10.221.128.1/api/v1/namespaces/rspsales-ci/persistentvolumeclaims
. Message: Forbidden!Configured service account doesn't have access.
Service account may have been revoked. persistentvolumeclaims
"pvc-b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j" is forbidden: cannot
set blockOwnerDeletion if an ownerReference refers to a resource you can't
set finalizers on: User "system:serviceaccount:rspsales-ci:jenkins" cannot
update pods/finalizers in project "rspsales-ci", .
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure
(OperationSupport.java:510)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.
assertResponseCode(OperationSupport.java:447)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse
(OperationSupport.java:413)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse
(OperationSupport.java:372)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(
OperationSupport.java:241)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(
BaseOperation.java:813)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(
BaseOperation.java:328)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(
BaseOperation.java:324)
at org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.
DynamicPVCWorkspaceVolume.createVolume(DynamicPVCWorkspaceVolume.java:94)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(
KubernetesLauncher.java:130)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:297)
at jenkins.util.ContextResettingExecutorService$2.call(
ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(
ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
ja)va:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:624)
at java.lang.Thread.run(Thread.java:748)
I guess it´s related to the Dynamic PVC´s (see JENKINS-47591) introduced in
1.19.2 - but how can this be resolved ?
The strange thing about it is that after restarting Jenkins the POD
launching works several times - and than suddenly starts to fail with the
above message.
I´m running Jenkins with a dedicated service-account:jenkins at openShift,
having either "edit" or now for testing "admin" role.
Thanx for any ideas,
Torsten
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/cdf14f70-981f-49e9-9b6a-16621910626c%40googlegroups.co
Re: Kubernetes plugin cannot start POD´s due to PVC creation error
I´ve enabled a "kubernetes" Logger with Level.FINEST and got this output:
Combining pod templates, parent: PodTemplate{inheritFrom='', name='default',
namespace='', label='', nodeSelector='', nodeUsageMode=EXCLUSIVE,
workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.
DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{name=
'main', image=
'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/jenkins-slave/base:latest'
, workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat',
resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='',
resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.
kubernetes.ContainerLivenessProbe@16be1b19}]} Oct 17, 2019 1:55:40 PM
FINEST org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils
Combining pod templates, template: PodTemplate{, name=
'9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', label=
'9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeUsageMode=EXCLUSIVE, containers
=[ContainerTemplate{name='main', image=
'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest'
, alwaysPullImage=true, command='/bin/sh -c', args='cat', ttyEnabled=true,
envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net,
getKey()=LOCAL_URL], KeyValueEnvVar
[getValue()=https://rspsales-cinext.mycompanygroup.net/nexus,
getKey()=NEXUS_URL]]}],
annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8,
org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
Oct 17, 2019 1:55:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes.
PodTemplateUtils
Pod templates combined: PodTemplate{inheritFrom='', name=
'9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', namespace='', label=
'9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeSelector='', nodeUsageMode=
EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.
workspace.DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{
name='main', image=
'docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest'
, alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh
-c', args='cat', ttyEnabled=true, resourceRequestCpu='',
resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='',
envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net,
getKey()=LOCAL_URL], KeyValueEnvVar
[getValue()=https://rspsales-cinext.mycompanygroup.net/nexus,
getKey()=NEXUS_URL]]}],
annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8,
org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=
I´m wondering why the parent PodTemplate has
workspaceVolume=DynamicPVCWorkspaceVolume?
Is this the new default?
I´ll try to enable a custom Workspace (EmptyDir) in Kubernetes plugin
configuration - which should prevent from creation of DynamicPVC´s causing the
error.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/8829441d-c93e-4546-bee2-b9d7358c7ebf%40googlegroups.com.
