dickens7 opened a new issue, #1412: URL: https://github.com/apache/apisix-ingress-controller/issues/1412
### Issue description An exception occurs when ApisixPluginConfig is used by multiple ApisixRoute or Ingress https://github.com/apache/apisix-ingress-controller/blob/v1.5.0/pkg/ingress/apisix_route.go#L329-L336 https://github.com/apache/apisix-ingress-controller/blob/v1.5.0/pkg/kube/translation/apisix_route.go#L924-L927 ### Environment - your apisix-ingress-controller version (output of apisix-ingress-controller version --long): Version: 1.5.0 Git SHA: no-git-module Go Version: go1.19.2 Building OS/Arch: linux/amd64 Running OS/Arch: linux/amd64 - your Kubernetes cluster version (output of kubectl version): v1.18.4+k3s1 - if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a): ### Minimal test code / Steps to reproduce 1. create ApisixPluginConfig ```yaml apiVersion: apisix.apache.org/v2 kind: ApisixPluginConfig metadata: name: cors-plugins spec: plugins: - name: cors enable: true ``` 2. create ApisixRoute `httpserver` and `httpserver2` ```yaml apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: httpserver spec: http: - name: httpserver match: hosts: - local.httpbin.org paths: - / backends: - serviceName: httpbin servicePort: 80 plugin_config_name: cors-plugins --- apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: httpserver2 spec: http: - name: httpserver match: hosts: - local.httpbin.org paths: - / backends: - serviceName: httpbin servicePort: 80 plugin_config_name: cors-plugins ``` ### Actual result ApisixPluginConfig will be deleted when ApisixRoute or Ingress is updated and deleted ### Error log ```shell 2022-10-27T18:20:23+08:00 info utils/manifest.go:361 plugin_config was referenced by other routes {"plugin_config_id": "9c092931", "plugin_config_name": ""} 2022-10-27T18:20:23+08:00 debug ingress/apisix_route.go:507 ApisixRoute update event arrived {"key": "xxx-env-dev/oa2-orgfront", "new object": {"metadata":{"name":"oa2-orgfront","namespace":"xxx-env-dev","selfLink":"/apis/apisix.apache.org/v2/namespaces/xxx-env-dev/apisixroutes/oa2-orgfront","uid":"2f2036ed-f973-4961-9534-3ce24627cec5","resourceVersion":"181495831","generation":4,"creationTimestamp":"2022-06-24T03:06:34Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"oa2-orgfront","meta.helm.sh/release-namespace":"xxx-env-dev"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}}},{"manager":"aslan","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.he lm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{}}},{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2022-10-27T10:20:22Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:http":{}},"f:status":{"f:conditions":{}}}}]},"spec":{"http":[{"name":"oa2-orgfront","match":{"paths":["/","/*"],"hosts":["org.k3s-jcyfb-dev.xxx.cn"]},"backends":[{"serviceName":"oa2-orgfront","servicePort":80,"weight":null}],"websocket":false,"plugin_config_name":"oa2-front-plugins","authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":4,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}, "old object": {"metadata":{"name":"oa2-orgfront","namespace":"xxx-env-dev","selfLink":"/apis/apisix.apache.org/v2/namespaces/xxx-env-dev/apisixroutes/oa 2-orgfront","uid":"2f2036ed-f973-4961-9534-3ce24627cec5","resourceVersion":"181495839","generation":4,"creationTimestamp":"2022-06-24T03:06:34Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"oa2-orgfront","meta.helm.sh/release-namespace":"xxx-env-dev"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}}},{"manager":"aslan","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{}}},{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2022-10-27T10:20:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec ":{"f:http":{}},"f:status":{"f:conditions":{}}}}]},"spec":{"http":[{"name":"oa2-orgfront","match":{"paths":["/","/*"],"hosts":["org.k3s-jcyfb-dev.xxx.cn"]},"backends":[{"serviceName":"oa2-orgfront","servicePort":80,"weight":null}],"websocket":false,"plugin_config_name":"oa2-front-plugins","authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":4,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}} 2022-10-27T18:20:23+08:00 debug apisix/pluginconfig.go:47 try to look up pluginConfig {"name": "xxx-env-dev_oa2-plugins", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugin_configs";, "cluster": "default"} 2022-10-27T18:20:23+08:00 debug ingress/apisix_route.go:304 translated ApisixRoute {"routes": [{"id":"1eedc198","name":"xxx-env-dev_oa2-org_oa2-org","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"hosts":["oa2.k3s-jcyfb-dev.xxx.cn"],"uris":["/org/api/","/org/api/*","/workflow/api/","/workflow/api/*"],"upstream_id":"f9f67824","plugin_config_id":"a7a986af"}], "upstreams": [{"id":"f9f67824","name":"xxx-env-dev_oa2-org_8888","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.42.1.156","port":8888,"weight":100}],"scheme":"http"}], "apisix_route": {}, "pluginConfigs": null} 2022-10-27T18:20:23+08:00 debug apisix/route.go:47 try to look up route {"name": "xxx-env-dev_oa2-org_oa2-org", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes";, "cluster": "default"} 2022-10-27T18:20:23+08:00 debug apisix/upstream.go:203 try to update upstream {"id": "f9f67824", "name": "xxx-env-dev_oa2-org_8888", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams"} 2022-10-27T18:20:23+08:00 debug apisix/upstreamservicerelation.go:95 try to create upstreamService in cache {"cluster": "default"} 2022-10-27T18:20:23+08:00 debug apisix/upstream.go:223 updating upstream {"body": "{\"id\":\"f9f67824\",\"name\":\"xxx-env-dev_oa2-org_8888\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"type\":\"roundrobin\",\"nodes\":[{\"host\":\"10.42.1.156\",\"port\":8888,\"weight\":100}],\"scheme\":\"http\"}", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/f9f67824"} 2022-10-27T18:20:23+08:00 debug apisix/resource.go:123 got upstream: {"pass_host":"pass","nodes":[{"port":8888,"host":"10.42.1.156","priority":0,"weight":100}],"hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","name":"xxx-env-dev_oa2-org_8888","update_time":1666866023,"labels":{"managed-by":"apisix-ingress-controller"},"id":"f9f67824","create_time":1656040035,"scheme":"http","type":"roundrobin"} 2022-10-27T18:20:23+08:00 debug apisix/route.go:205 try to update route {"id": "1eedc198", "name": "xxx-env-dev_oa2-org_oa2-org", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes"} 2022-10-27T18:20:23+08:00 debug apisix/route.go:219 updating route {"body": "{\"id\":\"1eedc198\",\"name\":\"xxx-env-dev_oa2-org_oa2-org\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"hosts\":[\"oa2.k3s-jcyfb-dev.xxx.cn\"],\"uris\":[\"/org/api/\",\"/org/api/*\",\"/workflow/api/\",\"/workflow/api/*\"],\"upstream_id\":\"f9f67824\",\"plugin_config_id\":\"a7a986af\"}", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/1eedc198"} 2022-10-27T18:20:23+08:00 debug apisix/resource.go:93 got route: {"hosts":["oa2.k3s-jcyfb-dev.xxx.cn"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"f9f67824","plugin_config_id":"a7a986af","update_time":1666866023,"create_time":1658209424,"priority":0,"labels":{"managed-by":"apisix-ingress-controller"},"id":"1eedc198","status":1,"uris":["\/org\/api\/","\/org\/api\/*","\/workflow\/api\/","\/workflow\/api\/*"],"name":"xxx-env-dev_oa2-org_oa2-org"} 2022-10-27T18:20:23+08:00 debug apisix/pluginconfig.go:180 try to delete pluginConfig {"id": "a7a986af", "name": "", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugin_configs"} 2022-10-27T18:20:23+08:00 error apisix/pluginconfig.go:191 failed to reflect pluginConfig delete to cache: still in use``` ### Expected result ApisixPluginConfig is not deleted when ApisixRoute or Ingress are updated and deleted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
