juzhiyuan commented on issue #2282: URL: https://github.com/apache/apisix-ingress-controller/issues/2282#issuecomment-3083325794
# Multi-Cluster Ingress APISIX Resource Isolation ## Background In APISIX Ingress Controller version 2.0.0+, the `namespaceSelector` has been deprecated and replaced with the recommendation to use `IngressClass` for resource isolation. This change is based on the following key reasons: ### 1. Kubernetes Official Standard Specification - Clearer and More Explicit `IngressClass` is the official standard mechanism introduced by Kubernetes starting from v1.18, used to explicitly specify which Controller manages specific resources. - **Resource-level declarative isolation**: Provides clear ownership declaration at the resource level - **No dependency on controller-specific behavior**: Works independently without requiring special controller implementations - **Direct declaration**: Can be declared directly on Ingress, Gateway, and other resources for easy use and maintenance In contrast, `namespaceSelector` is a behavior supported by some Ingress Controller implementations (not all controllers support it) and is not a core Kubernetes mechanism. ### 2. Precise Granularity - Avoiding Mishandling `namespaceSelector` performs resource filtering at the namespace level. Once a namespace is selected, it processes all Ingress resources within that namespace without the ability to further refine the selection. `ingressClassName` provides precise control at the individual resource level with finer granularity, avoiding the processing of resources that shouldn't be handled. ### 3. Better Cross-Namespace Support If you want an Ingress Controller to manage Ingress resources across multiple namespaces, you simply need to set the corresponding `ingressClassName` on those Ingress resources. - **IngressClass supports cross-namespace operations** - **namespaceSelector requires maintaining additional label strategies with inflexible control scope** ### 4. Unified with Gateway API and Other Mechanisms Beyond just Ingress, Kubernetes Gateway API, IngressClass, and GatewayClass all adopt similar mechanisms, using Class mechanisms to specify controller behavior. Using IngressClass aligns better with official architecture design. The use of namespaceSelector with the Gateway API's ReferenceGrant may create conflicts or trigger inconsistent behavior: https://gateway-api.sigs.k8s.io/api-types/referencegrant/ ### 5. Easier Debugging and Understanding When developers or platform administrators see `ingressClassName=pro-apisix` explicitly specified in resources, they can clearly understand which controller manages the resource. In contrast, the effects of `namespaceSelector` are implicit and not intuitive. ## How to Deploy and Use Multiple Ingress-APISIX Clusters in the Same Cluster In actual production or multi-tenant environments, it's common to deploy multiple independent Ingress APISIX instances in the same Kubernetes cluster to achieve traffic isolation, environment separation (such as testing, staging, production), or service team autonomy. This document covers how to deploy and use multiple ingress-apisix clusters in parallel within the same cluster, including: - How to install multiple independent ingress-apisix instances using Helm - How to isolate traffic resources using IngressClass - How to configure separate routing rules for each instance - How to verify the independence and correctness of each Ingress APISIX instance through testing ## Installation and Deployment ### Important Helm Chart Deployment Parameters | Parameter | Description | Purpose | |-----------|-------------|---------| | `--set ingress-controller.config.kubernetes.ingressClass` | IngressClass name | Specify the name of the IngressClass managed by this Ingress Controller as `pro-apisix`. the Ingress resources in Kubernetes that specify this name will be processed by this Controller. | | `--set ingress-controller.gatewayProxy.createDefault` | Create default GatewayProxy | Configure the default GatewayProxy (used to describe how the Controller accesses the APISIX Admin API for use by the Ingress Controller IngressClass, ). | | `--set ingress-controller.config.controllerName` | Controller unique identifier | Distinguishes different controller instances | | `--set ingress-controller.apisix.adminService.name` | Admin service name | Avoids naming conflicts | ### Deploy Test Environment Cluster (test-apisix) ```bash helm repo add apisix https://charts.apiseven.com helm repo update helm install test-apisix \ --namespace test-ingress-apisix \ --create-namespace \ --set apisix.deployment.role=traditional \ --set apisix.deployment.role_traditional.config_provider=yaml \ --set etcd.enabled=false \ --set ingress-controller.enabled=true \ --set ingress-controller.config.kubernetes.ingressClass=test-apisix \ --set ingress-controller.config.provider.type=apisix-standalone \ --set ingress-controller.gatewayProxy.createDefault=true \ --set ingress-controller.config.controllerName=apisix.apache.org/apisix-ingress-controller/test \ --set ingress-controller.apisix.adminService.name=test-apisix-admin \ apisix/apisix ``` After successful installation, Helm will create the following resources: ```yaml apiVersion: apisix.apache.org/v1alpha1 kind: GatewayProxy metadata: name: test-apisix-config namespace: test-ingress-apisix spec: provider: controlPlane: auth: adminKey: value: edd1c9f034335f136f87ad84b625c8f1 type: AdminKey service: name: test-apisix-admin port: 9180 type: ControlPlane --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: test-apisix spec: controller: apisix.apache.org/apisix-ingress-controller/test parameters: apiGroup: apisix.apache.org kind: GatewayProxy name: test-apisix-config namespace: test-ingress-apisix scope: Namespace ``` ### Deploy Production Environment Cluster (pro-apisix) ```bash helm repo add apisix https://charts.apiseven.com helm repo update helm install pro-apisix \ --namespace pro-ingress-apisix \ --create-namespace \ --set apisix.deployment.role=traditional \ --set apisix.deployment.role_traditional.config_provider=yaml \ --set etcd.enabled=false \ --set ingress-controller.enabled=true \ --set ingress-controller.config.kubernetes.ingressClass=pro-apisix \ --set ingress-controller.config.provider.type=apisix-standalone \ --set ingress-controller.gatewayProxy.createDefault=true \ --set ingress-controller.config.controllerName=apisix.apache.org/apisix-ingress-controller/pro \ --set ingress-controller.apisix.adminService.name=pro-apisix-admin \ apisix/apisix ``` After successful installation, Helm will create the following resources: ```yaml apiVersion: apisix.apache.org/v1alpha1 kind: GatewayProxy metadata: name: pro-apisix-config namespace: pro-ingress-apisix spec: provider: controlPlane: auth: adminKey: value: edd1c9f034335f136f87ad84b625c8f1 type: AdminKey service: name: pro-apisix-admin port: 9180 type: ControlPlane --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: pro-apisix spec: controller: apisix.apache.org/apisix-ingress-controller/pro parameters: apiGroup: apisix.apache.org kind: GatewayProxy name: pro-apisix-config namespace: pro-ingress-apisix scope: Namespace ``` ## Route Configuration ### Production Environment Route By specifying `ingressClassName: pro-apisix`, this resource will be managed by the production environment's pro-apisix Controller: ```yaml apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: pro-route-1 spec: ingressClassName: pro-apisix http: - name: rule0 match: hosts: - httpbin-pro paths: - /* backends: - serviceName: httpbin servicePort: 80 ``` ### Test Environment Route By specifying `ingressClassName: test-apisix`, this resource will be managed by the test environment's test-apisix Controller: ```yaml apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: test-route-1 spec: ingressClassName: test-apisix http: - name: rule0 match: hosts: - httpbin-test paths: - /* backends: - serviceName: httpbin servicePort: 80 ``` ## Verification ### Port Forwarding Setup Forward production environment APISIX to local port 9081: ```bash kubectl port-forward -n pro-ingress-apisix svc/pro-apisix-gateway 9081:80 ``` Forward test environment APISIX to local port 9082: ```bash kubectl port-forward -n test-ingress-apisix svc/test-apisix-gateway 9082:80 ``` ### Test Production Environment APISIX ```bash # Test with test host (should fail - route not found) ➜ curl localhost:9081/get -H "Host: httpbin-test" {"error_msg":"404 Route Not Found"} # Test with production host (should succeed) ➜ curl localhost:9081/get -H "Host: httpbin-pro" { "args": {}, "headers": { "Accept": "*/*", "Host": "httpbin-pro", "User-Agent": "curl/7.81.0", "X-Forwarded-Host": "httpbin-pro" }, "origin": "127.0.0.1", "url": "http://httpbin-pro/get"; } ``` ### Test Test Environment APISIX ```bash # Test with production host (should fail - route not found) ➜ curl localhost:9082/get -H "Host: httpbin-pro" {"error_msg":"404 Route Not Found"} # Test with test host (should succeed) ➜ curl localhost:9082/get -H "Host: httpbin-test" { "args": {}, "headers": { "Accept": "*/*", "Host": "httpbin-test", "User-Agent": "curl/7.81.0", "X-Forwarded-Host": "httpbin-test" }, "origin": "127.0.0.1", "url": "http://httpbin-test/get"; } ``` ## Conclusion The test results demonstrate that: 1. **Perfect Isolation**: Each APISIX instance only responds to routes configured with its corresponding IngressClass 2. **Cross-Environment Protection**: Production environment only handles `httpbin-pro` requests, while test environment only handles `httpbin-test` requests 3. **Independent Operation**: Both environments operate independently without interference This approach provides a robust solution for multi-tenant or multi-environment deployments within a single Kubernetes cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
