[ https://issues.apache.org/jira/browse/SUBMARINE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
cdmikechen reassigned SUBMARINE-1124: ------------------------------------- Assignee: cdmikechen > The automatic deletion of notebook does not delete the related PVC and > IngressRoute > ----------------------------------------------------------------------------------- > > Key: SUBMARINE-1124 > URL: https://issues.apache.org/jira/browse/SUBMARINE-1124 > Project: Apache Submarine > Issue Type: Bug > Components: K8s Submitter > Reporter: cdmikechen > Assignee: cdmikechen > Priority: Major > Labels: pull-request-available > Fix For: 0.7.0 > > Attachments: image-2021-12-10-09-56-32-491.png, screenshot-1.png > > > I use newest submarine server to create a notebook called *test*. Notebook > pod can not start by this error with image > *apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT* > !image-2021-12-10-09-56-32-491.png! > Submarine will consistently display the status of the notebook in waiting. > !screenshot-1.png! > After a long time, submarine will automatically delete the notebook row in > the database. I see the corresponding log information in the background. > {code} > [INFO ] 2021-12-10 09:23:25,015 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Checked out connection 1073981532 from pool. > Testing connection 1073981532 ... > Connection 1073981532 is GOOD! > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c] > Testing connection 1073981532 ... > Connection 1073981532 is GOOD! > Returned connection 1073981532 to pool. > [INFO ] 2021-12-10 09:23:25,593 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:25,602 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [INFO ] 2021-12-10 09:23:25,904 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:25,918 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [WARN ] 2021-12-10 09:23:27,257 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [INFO ] 2021-12-10 09:23:27,279 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Checked out connection 1073981532 from pool. > Testing connection 1073981532 ... > [INFO ] 2021-12-10 09:23:27,283 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Connection 1073981532 is GOOD! > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c] > [WARN ] 2021-12-10 09:23:28,972 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [INFO ] 2021-12-10 09:23:29,221 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > [INFO ] 2021-12-10 09:23:30,257 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > [INFO ] 2021-12-10 09:23:31,273 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > [INFO ] 2021-12-10 09:23:32,683 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Created connection 1441218336. > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@55e73f20] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > Created connection 875758614. > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@34330416] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > Created connection 616832400. > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@24c41d90] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > Created connection 670559153. > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@27f7ebb1] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > Created connection 2081921914. > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@7c179b7a] > Testing connection 1073981532 ... > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > Connection 1073981532 is GOOD! > Returned connection 1073981532 to pool. > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@34330416] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416] > Testing connection 875758614 ... > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@24c41d90] > Connection 875758614 is GOOD! > Returned connection 875758614 to pool. > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@27f7ebb1] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90] > Testing connection 616832400 ... > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1] > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@7c179b7a] > <== Columns: id, notebook_spec, create_by, create_time, update_by, > update_time > <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 > 09:33:27.0, null, 2021-12-09 09:33:27.0 > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a] > Connection 616832400 is GOOD! > Returned connection 616832400 to pool. > Testing connection 2081921914 ... > <== Total: 1 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@55e73f20] > Connection 2081921914 is GOOD! > Returned connection 2081921914 to pool. > Testing connection 670559153 ... > Connection 670559153 is GOOD! > Returned connection 670559153 to pool. > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20] > Testing connection 1441218336 ... > <== Updates: 1 > Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145] > <== Updates: 0 > Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f] > Connection 1441218336 is GOOD! > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@41c15145] > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@124f309f] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145] > Closed connection 1441218336. > Testing connection 1103188293 ... > Connection 1103188293 is GOOD! > Closed connection 1103188293. > Testing connection 307179679 ... > [WARN ] 2021-12-10 09:23:35,350 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > Connection 307179679 is GOOD! > Closed connection 307179679. > [WARN ] 2021-12-10 09:23:35,352 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [INFO ] 2021-12-10 09:23:38,326 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:38,334 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [INFO ] 2021-12-10 09:23:38,338 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:38,342 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [INFO ] 2021-12-10 09:23:40,057 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:40,062 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [INFO ] 2021-12-10 09:23:42,452 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:42,458 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [INFO ] 2021-12-10 09:23:42,705 > method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62) > Upstream response JSON: > {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda > -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 > \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | > tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should > be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; > exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | > cut -f2 -d\u0027 \u0027);. Moving forward with env creation and > activation.\"; > fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off > 2m40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off > 1m20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off > 40s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off > 20s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off > 10s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off > 5m0s restarting failed container\u003dtest > pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}} > [INFO ] 2021-12-10 09:23:42,711 > method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457) > notebook status: waiting; check the pods in namespace:[submarine] to ensure > is the waiting caused by image pulling > [WARN ] 2021-12-10 09:23:43,055 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [WARN ] 2021-12-10 09:23:43,065 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [INFO ] 2021-12-10 09:23:43,078 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Checked out connection 1073981532 from pool. > Testing connection 1073981532 ... > Connection 1073981532 is GOOD! > Setting autocommit to false on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, > update_time from notebook > ==> Parameters: > <== Total: 0 > Resetting autocommit to true on JDBC Connection > [com.mysql.jdbc.JDBC4Connection@4003a85c] > Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c] > Testing connection 1073981532 ... > Connection 1073981532 is GOOD! > Returned connection 1073981532 to pool. > [WARN ] 2021-12-10 09:23:43,083 > method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176) > Unsupported parse class java.util.ArrayList Dict Annotation! > [INFO ] 2021-12-10 09:23:43,085 > method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43) > Notebook selectAll > Opening JDBC Connection > Checked out connection 875758614 from pool. > {code} > I checked codes and found that submarine just delete database row but it does > not delete the resource like CRD, PVC and IngressRoute. So that if I recreate > with a duplicate name, submarine tell me I can not create this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org