[ https://issues.apache.org/jira/browse/SUBMARINE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
cdmikechen reassigned SUBMARINE-1154: ------------------------------------- Assignee: cdmikechen > tf-job-operator can not create experiment pod in openshift > ---------------------------------------------------------- > > Key: SUBMARINE-1154 > URL: https://issues.apache.org/jira/browse/SUBMARINE-1154 > Project: Apache Submarine > Issue Type: Bug > Components: experiment > Reporter: cdmikechen > Assignee: cdmikechen > Priority: Critical > Labels: pull-request-available > Fix For: 0.7.0 > > Attachments: image-2021-12-21-16-42-30-133.png > > > I use OKD 4.7 to run submarine. When I created an experiment, I can not find > pod started successfully. > I checked tf-job-operator log and found that some problems were encountered > when operator creating pod. > > {code} > {"filename":"app/server.go:76","level":"info","msg":"EnvKubeflowNamespace not > set, use default namespace","time":"2021-12-21T08:17:42Z"} > {"filename":"app/server.go:80","level":"info","msg":"Using cluster scoped > operator","time":"2021-12-21T08:17:42Z"} > {"filename":"app/server.go:86","level":"info","msg":"[API Version: v1 > Version: v0.1.0-alpha Git SHA: Not provided. Go Version: go1.13.5 Go OS/Arch: > linux/amd64]","time":"2021-12-21T08:17:42Z"} > W1221 08:17:42.318664 1 client_config.go:548] Neither --kubeconfig nor > --master was specified. Using the inClusterConfig. This might not work. > {"filename":"tf-operator.v1/main.go:42","level":"info","msg":"Setting up > client for monitoring on port: 8443","time":"2021-12-21T08:17:42Z"} > {"filename":"tensorflow/controller.go:120","level":"info","msg":"Creating > TFJob controller","time":"2021-12-21T08:17:42Z"} > {"filename":"tensorflow/controller.go:127","level":"info","msg":"Creating Job > controller","time":"2021-12-21T08:17:42Z"} > I1221 08:17:42.377440 1 leaderelection.go:187] attempting to acquire > leader lease default/tf-operator... > {"filename":"tensorflow/job.go:94","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"TFJob > experiment-1640073634042-0001 is > created.","time":"2021-12-21T08:17:42Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > I1221 08:18:08.422211 1 leaderelection.go:196] successfully acquired > lease default/tf-operator > {"filename":"tensorflow/controller.go:187","level":"info","msg":"Starting > TFJob controller","time":"2021-12-21T08:18:08Z"} > {"filename":"tensorflow/controller.go:190","level":"info","msg":"Waiting for > informer caches to sync","time":"2021-12-21T08:18:08Z"} > {"filename":"tensorflow/controller.go:196","level":"info","msg":"Starting 1 > workers","time":"2021-12-21T08:18:08Z"} > {"filename":"tensorflow/controller.go:202","level":"info","msg":"Started > workers","time":"2021-12-21T08:18:08Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:08Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:08Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (859.732664ms)","time":"2021-12-21T08:18:09Z"} > E1221 08:18:09.383188 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (271.610488ms)","time":"2021-12-21T08:18:09Z"} > E1221 08:18:09.664630 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (16.622312ms)","time":"2021-12-21T08:18:09Z"} > E1221 08:18:09.695125 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (20.935691ms)","time":"2021-12-21T08:18:09Z"} > E1221 08:18:09.742111 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (126.441784ms)","time":"2021-12-21T08:18:09Z"} > E1221 08:18:09.909122 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (37.721013ms)","time":"2021-12-21T08:18:10Z"} > E1221 08:18:10.027525 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (36.386638ms)","time":"2021-12-21T08:18:10Z"} > E1221 08:18:10.224908 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (172.821236ms)","time":"2021-12-21T08:18:10Z"} > E1221 08:18:10.753307 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (73.171981ms)","time":"2021-12-21T08:18:11Z"} > E1221 08:18:11.469567 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:11Z"} > {"filename":"tensorflow/job.go:130","level":"info","msg":"Updating tfjob: > experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (36.756625ms)","time":"2021-12-21T08:18:12Z"} > E1221 08:18:12.423556 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:12Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (24.93148ms)","time":"2021-12-21T08:18:12Z"} > E1221 08:18:12.785557 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:12Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (22.903888ms)","time":"2021-12-21T08:18:17Z"} > E1221 08:18:17.945442 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:17Z"} > {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile > TFJobs > experiment-1640073634042-0001","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need > to create new pod: > worker-0","replica-type":"worker","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods > error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot > set blockOwnerDeletion if an ownerReference refers to a resource you can't > set finalizers on: , > \u003cnil\u003e","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"} > {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished > syncing tfjob \"submarine/experiment-1640073634042-0001\" > (183.401302ms)","time":"2021-12-21T08:18:28Z"} > E1221 08:18:28.381347 1 controller.go:266] error syncing tfjob: pods > "experiment-1640073634042-0001-worker-0" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , <nil> > {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\", > Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\", > UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\", > ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason: > 'FailedCreatePod' Error creating: pods > \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set > blockOwnerDeletion if an ownerReference refers to a resource you can't set > finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:28Z"} > {code} > I checked training-operator and search *finalizers* > https://github.com/kubeflow/training-operator/search?q=finalizers > !image-2021-12-21-16-42-30-133.png! > The screenshot should prove that we need to supplement relevant resources > when RBAC is created. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org