yj-lee0503 opened a new issue, #3465: URL: https://github.com/apache/polaris/issues/3465
### Is your feature request related to a problem? Please describe. Yes. Apache Polaris 1.2.0-incubating and 1.3.0-incubating do not support HTTP proxies for federated external catalog connections, making federation incompatible with controlled egress environments where all outbound traffic must flow through a proxy server. ## The Problem: - Organizations with controlled egress policies (common in enterprise and regulated environments) cannot use Polaris federation features - HTTP traffic cannot be routed through corporate proxies (Squid, Nginx, etc.) - Network policies that block direct internet access prevent federation from working - Proxy configuration properties are set but completely ignored by the current Iceberg version ## Root Cause: - Apache Iceberg added HTTP proxy support on May 7, 2025 via [PR #12406](https://github.com/apache/iceberg/pull/12406) - However, this feature has not been included in any Iceberg release yet (latest is 1.10.1, Dec 2025) - Polaris 1.2.0 uses Iceberg 1.10.1, and Polaris 1.3.0 uses Iceberg 1.10.0 - Proxy support exists only in Iceberg's main branch (unreleased) ## Testing Evidence: We extensively tested all proxy configuration methods and confirmed none work with current Polaris versions: 1. ✅ Configured rest.client.proxy.* properties in catalog definition - Ignored 1. ✅ Set environment variables (HTTPS_PROXY, HTTP_PROXY, etc.) - Not used 1. ✅ Set JVM proxy properties (-Dhttps.proxyHost, -Dhttp.proxyPort) - Not used 1. ✅ Set Quarkus REST client config (QUARKUS_REST_CLIENT_PROXY_ADDRESS) - Not used 1. ✅ Applied iptables DNAT rules for transparent proxy - Partial workaround only 1. ❌ Result: Zero proxy traffic in logs, federation fails in isolated networks ### Describe the solution you'd like When Apache Iceberg releases version 1.11.0 (or the next version) with proxy support included, upgrade Polaris's Iceberg dependency to that version. ## Specific Change: - File: gradle/libs.versions.toml - Line 23: Change iceberg = "1.10.1" → iceberg = "1.11.0" (or latest with proxy support) **Expected Behavior After Upgrade:** Users would be able to configure HTTP proxy for federation via catalog properties when creating external catalogs: ``` { "type": "EXTERNAL", "name": "my-external-catalog", "properties": { "rest.client.proxy.hostname": "proxy.example.com", "rest.client.proxy.port": "3128", "rest.client.proxy.username": "optional-username", "rest.client.proxy.password": "optional-password", "rest.client.connection-timeout-ms": "30000", "rest.client.socket-timeout-ms": "120000" }, "connectionConfigInfo": { "connectionType": "ICEBERG_REST", "uri": "https://remote-catalog.example.com/api/catalog/v1";, "remoteCatalogName": "remote-catalog", "authenticationParameters": { ... } } } ``` ## Benefits: - Enables federation in enterprise environments with controlled egress - Improves security by allowing traffic inspection and logging - Maintains compatibility with existing network policies - Also brings performance improvements, bug fixes, and enhanced REST catalog features from newer Iceberg versions ### Describe alternatives you've considered All current workarounds are suboptimal: 1. **Deploy in NAT-enabled nodes** - Works but reduces network security posture by allowing direct internet access 1. **Use AWS PrivateLink/VPC endpoints** - Only available for specific cloud providers, specific service tiers, and specific federation targets (e.g., Snowflake Business Critical edition) 1. **iptables transparent proxy** - Complex, brittle, requires NET_ADMIN capability, and limited by DNS round-robin (only captures IPs resolved at init time) 1. **Build custom Polaris** - Use Iceberg main branch snapshot - Not recommended for production due to unreleased code None of these alternatives provide a clean, production-ready solution for general HTTP proxy support. ### Additional context ## Environment: - Polaris Version: 1.2.0-incubating (also tested 1.3.0-incubating) - Iceberg Version (current): 1.10.1 (latest release as of Dec 2025) - Iceberg Version (with proxy support): main branch only (unreleased) - Deployment: Kubernetes (EKS) - Network: Controlled egress via Squid proxy - Federation Target: Snowflake Open Catalog ##References: - [Apache Iceberg Issue #9174 - Proxy Support Unavailable](https://github.com/apache/iceberg/issues/9174) - [Apache Iceberg PR #12406 - Enable HTTP Proxy Support](https://github.com/apache/iceberg/pull/12406) - [Iceberg Commit Notification - Proxy Support](https://www.mail-archive.com/[email protected]/msg16074.html) - [Iceberg 1.10.1 Release](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.10.1) ## Acceptance Criteria: - [ ] Apache Iceberg releases version 1.11.0+ with proxy support - [ ] Polaris upgrades to Iceberg 1.11.0 or later - [ ] Proxy configuration properties documented in Polaris docs - [ ] Federation with proxy configuration tested in CI - [ ] Release notes mention proxy support ## Impact Assessment: - Impact: High - Blocks federation adoption in enterprise environments - Effort: I don't know -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
