This is an automated email from the ASF dual-hosted git repository.
diqiu50 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 03da573642 [#13098] improvement(server): Support an advertised URI for
Iceberg REST service discovery (#13126)
03da573642 is described below
commit 03da573642dcee1a97e896b0fa9d3e86a0b7315e
Author: Yuhui <[email protected]>
AuthorDate: Tue Sep 15 14:16:00 2026 +0800
[#13098] improvement(server): Support an advertised URI for Iceberg REST
service discovery (#13126)
### What changes were proposed in this pull request?
- Add `gravitino.iceberg-rest.advertised-uri`. When set,
`/api/system/iceberg-rest` reports it as-is after the existing service
availability, config provider and metalake checks; otherwise the
endpoint is derived from the listener configuration as before.
- Validate the configured value (absolute `http`/`https` URI with a
host, no query or fragment). An invalid value fails the discovery
request with an internal error instead of being silently ignored.
- Document `/system/iceberg-rest` in the OpenAPI spec and describe the
new property in the Iceberg REST service and Trino connector docs.
### Why are the changes needed?
Behind a reverse proxy the public Iceberg REST endpoint may differ from
the listener's scheme, host, port or path, so discovery reports an
endpoint clients cannot reach and every Trino client has to set
`gravitino.iceberg.rest-uri` manually.
Fix: #13098
### Does this PR introduce _any_ user-facing change?
New optional server property `gravitino.iceberg-rest.advertised-uri`. It
only affects the advertised endpoint; the listener configuration is
unchanged.
### How was this patch tested?
Unit tests in `TestIcebergRESTServiceOperations` cover the reverse-proxy
endpoint, fallback to the derived endpoint, metalake and config provider
checks, and invalid values. `./gradlew :docs:build` validates the
OpenAPI spec.
---
docs/iceberg-rest-service.md | 6 ++
docs/open-api/openapi.yaml | 3 +
docs/open-api/system.yaml | 95 ++++++++++++++++++++++
docs/trino-connector/catalog-iceberg.md | 6 ++
.../web/rest/IcebergRESTServiceOperations.java | 45 +++++++++-
.../web/rest/TestIcebergRESTServiceOperations.java | 91 +++++++++++++++++++++
6 files changed, 245 insertions(+), 1 deletion(-)
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index 76f35650db..098c6a7feb 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -125,10 +125,16 @@ Do not add them to the standalone server configuration.
| `gravitino.iceberg-rest.responseHeaderSize` | The maximum size of an
HTTP response.
| `131072`
| No |
| `gravitino.iceberg-rest.includeErrorStackTrace` | Whether error responses
include server-side stack traces. Set this to `false` in new deployments
because responses can expose internal implementation details. | `true`
| No |
| `gravitino.iceberg-rest.customFilters` | Comma-separated list of
filter class names to apply to the APIs.
| (none)
| No |
+| `gravitino.iceberg-rest.advertised-uri` | The public endpoint
reported to clients that discover the service through the Gravitino server.
Must be an absolute `http`/`https` URI with a host and no query or fragment.
| (none)
| No |
The filter in `customFilters` should be a standard javax servlet filter.
Specify filter parameters by setting configuration entries in the style
`gravitino.iceberg-rest.<class name of filter>.param.<param name>=<value>`.
+Set `gravitino.iceberg-rest.advertised-uri` when clients such as the Trino
connector reach the service through a reverse proxy whose scheme, host, port or
path differs from the listener's, for example
`https://iceberg.example.com/iceberg/`.
+An explicit port must be in the range 1-65535; an invalid value makes
discovery requests fail instead of advertising an unreachable endpoint.
+It only affects the advertised endpoint, not the listener; when unset, the
endpoint is derived from `host`, `httpPort`/`httpsPort` and `enableHttps`.
+This setting applies only to the auxiliary service.
+
#### Asynchronous Table Purge
By default, dropping a table with `purgeRequested=true` is synchronous: the
catalog entry and the table files are removed before the `DELETE` returns.
diff --git a/docs/open-api/openapi.yaml b/docs/open-api/openapi.yaml
index 826dd33aa8..5af0739eb1 100644
--- a/docs/open-api/openapi.yaml
+++ b/docs/open-api/openapi.yaml
@@ -58,6 +58,9 @@ paths:
/health/ready:
$ref: "./health.yaml#/paths/~1health~1ready"
+ /system/iceberg-rest:
+ $ref: "./system.yaml#/paths/~1system~1iceberg-rest"
+
/metalakes:
# escape the path name by replacing `/` with `~1`
$ref: "./metalakes.yaml#/paths/~1metalakes"
diff --git a/docs/open-api/system.yaml b/docs/open-api/system.yaml
new file mode 100644
index 0000000000..6558c10b90
--- /dev/null
+++ b/docs/open-api/system.yaml
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+
+paths:
+ /system/iceberg-rest:
+ get:
+ tags:
+ - system
+ summary: Discover the Iceberg REST service endpoint
+ operationId: getIcebergRestServiceUri
+ description: >
+ Reports the endpoint of the Iceberg REST service running as an
auxiliary service of this
+ Gravitino server, so that clients which already connect to the
Gravitino server, such as
+ the Trino connector, can discover it instead of configuring it
separately.
+ The reported `uri` is `null` when the service is not registered, does
not use the dynamic
+ catalog config provider, or serves a different metalake than the
requested one.
+ When `gravitino.iceberg-rest.advertised-uri` is configured, it is
reported as-is;
+ otherwise the endpoint is derived from the service's listener
configuration, and a wildcard
+ listener host is replaced with the host the caller used to reach the
Gravitino server.
+ The response is therefore marked `Cache-Control: no-store`.
+ parameters:
+ - name: metalake
+ in: query
+ description: >
+ The metalake the caller intends to route through the Iceberg REST
service. When
+ omitted or blank, the endpoint is reported regardless of which
metalake it serves.
+ required: false
+ schema:
+ type: string
+ responses:
+ "200":
+ $ref: "#/components/responses/IcebergRESTServiceResponse"
+ "4XX":
+ $ref: "./openapi.yaml#/components/responses/BadRequestErrorResponse"
+ "5XX":
+ $ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+components:
+ responses:
+ IcebergRESTServiceResponse:
+ description: The Iceberg REST service endpoint
+ content:
+ application/vnd.gravitino.v1+json:
+ schema:
+ $ref: "#/components/schemas/IcebergRESTServiceResponse"
+ examples:
+ IcebergRESTServiceAvailable:
+ $ref: "#/components/examples/IcebergRESTServiceAvailable"
+ IcebergRESTServiceUnavailable:
+ $ref: "#/components/examples/IcebergRESTServiceUnavailable"
+
+ schemas:
+ IcebergRESTServiceResponse:
+ type: object
+ required:
+ - code
+ properties:
+ code:
+ type: integer
+ description: Response code, 0 for success
+ example: 0
+ uri:
+ type: string
+ nullable: true
+ description: >
+ The Iceberg REST service endpoint, or null when the service is not
running or does not
+ serve the requested metalake
+ example: "http://gravitino-host:9001/iceberg";
+
+ examples:
+ IcebergRESTServiceAvailable:
+ value:
+ code: 0
+ uri: "http://gravitino-host:9001/iceberg";
+
+ IcebergRESTServiceUnavailable:
+ value:
+ code: 0
+ uri: null
diff --git a/docs/trino-connector/catalog-iceberg.md
b/docs/trino-connector/catalog-iceberg.md
index 6244590e72..ccfae7bf9e 100644
--- a/docs/trino-connector/catalog-iceberg.md
+++ b/docs/trino-connector/catalog-iceberg.md
@@ -48,6 +48,12 @@ as part of that catalog's own definition — the same way
Trino replicates any o
cluster-wide. A catalog that could not be registered before the IRC started is
registered
automatically after a later discovery poll succeeds; no Trino restart is
required.
+The Gravitino server derives the discovered endpoint from the IRC's listener
configuration, so
+behind a reverse proxy it may report an endpoint clients cannot reach. In that
case, set
+`gravitino.iceberg-rest.advertised-uri` on the Gravitino server to the public
endpoint (see
+[Iceberg REST service](../iceberg-rest-service.md#http-server)); discovery
then reports that URI
+instead.
+
Set `gravitino.iceberg.rest-uri` to override the discovered endpoint, and it
is required — not just
an override — for a standalone IRC (its own process, not the Gravitino
server's auxiliary service):
the Gravitino server has no way to know a standalone IRC exists, so discovery
never finds one. See
diff --git
a/server/src/main/java/org/apache/gravitino/server/web/rest/IcebergRESTServiceOperations.java
b/server/src/main/java/org/apache/gravitino/server/web/rest/IcebergRESTServiceOperations.java
index 05969db42f..ce10a056e4 100644
---
a/server/src/main/java/org/apache/gravitino/server/web/rest/IcebergRESTServiceOperations.java
+++
b/server/src/main/java/org/apache/gravitino/server/web/rest/IcebergRESTServiceOperations.java
@@ -20,6 +20,8 @@ package org.apache.gravitino.server.web.rest;
import com.codahale.metrics.annotation.ResponseMetered;
import com.codahale.metrics.annotation.Timed;
+import java.net.URI;
+import java.net.URISyntaxException;
import java.util.Map;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.Consumes;
@@ -63,6 +65,8 @@ public class IcebergRESTServiceOperations {
// The post-strip key used by the Iceberg REST server itself; see
// IcebergConstants.GRAVITINO_METALAKE and DynamicIcebergConfigProvider.
private static final String SERVED_METALAKE_KEY = "gravitino-metalake";
+ // Overrides the listener-derived endpoint; see docs/iceberg-rest-service.md.
+ private static final String ADVERTISED_URI_KEY = "advertised-uri";
private static final String HOST_KEY = "host";
private static final String HTTP_PORT_KEY = "httpPort";
private static final String HTTPS_PORT_KEY = "httpsPort";
@@ -90,9 +94,18 @@ public class IcebergRESTServiceOperations {
@Timed(name = "iceberg-rest-service." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "iceberg-rest-service", absolute = true)
public Response getIcebergRestServiceUri(@QueryParam("metalake") String
metalake) {
+ String uri;
+ try {
+ uri = resolveUri(metalake);
+ } catch (IllegalStateException e) {
+ // A misconfiguration, re-reported on every discovery poll until fixed;
the message alone
+ // identifies it, so the stack trace is omitted from both the log and
the response.
+ LOG.error("Failed to resolve the Iceberg REST service endpoint: {}",
e.getMessage());
+ return Utils.internalError(e.getMessage());
+ }
// The reported host can depend on the caller's own Host header (see
resolveUri), so this
// response must never be cached and replayed to a different caller.
- return Response.fromResponse(Utils.ok(new
IcebergRESTServiceResponse(resolveUri(metalake))))
+ return Response.fromResponse(Utils.ok(new IcebergRESTServiceResponse(uri)))
.header("Cache-Control", "no-store")
.build();
}
@@ -147,6 +160,11 @@ public class IcebergRESTServiceOperations {
return null;
}
+ String advertisedUri =
StringUtils.trimToNull(config.get(ADVERTISED_URI_KEY));
+ if (advertisedUri != null) {
+ return checkAdvertisedUri(advertisedUri);
+ }
+
String host = config.getOrDefault(HOST_KEY, DEFAULT_HOST);
if (isWildcardHost(host)) {
// The Iceberg REST server binds to all interfaces, so it has no single
externally
@@ -166,6 +184,31 @@ public class IcebergRESTServiceOperations {
return String.format("%s://%s:%d/iceberg", scheme, bracketIfIPv6(host),
port);
}
+ private static String checkAdvertisedUri(String value) {
+ boolean valid;
+ try {
+ URI uri = new URI(value);
+ // URI accepts any non-negative integer as a port; -1 means no explicit
port.
+ int port = uri.getPort();
+ valid =
+ StringUtils.equalsAnyIgnoreCase(uri.getScheme(), "http", "https")
+ && StringUtils.isNotBlank(uri.getHost())
+ && (port == -1 || (port >= 1 && port <= 65535))
+ && uri.getQuery() == null
+ && uri.getFragment() == null;
+ } catch (URISyntaxException e) {
+ valid = false;
+ }
+ if (!valid) {
+ throw new IllegalStateException(
+ String.format(
+ "Invalid Iceberg REST service %s '%s': expected an absolute
http(s) URI with a "
+ + "host, a port in 1-65535 if present, and no query or
fragment",
+ ADVERTISED_URI_KEY, value));
+ }
+ return value;
+ }
+
// An IPv6 literal host (e.g. "::1", from an explicit config value or from
// HttpServletRequest#getServerName()) must be bracketed to form a valid URI
authority;
// otherwise its colons are parsed as the port separator. A hostname or IPv4
address never
diff --git
a/server/src/test/java/org/apache/gravitino/server/web/rest/TestIcebergRESTServiceOperations.java
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestIcebergRESTServiceOperations.java
index 3db9f8fa43..ef34808f55 100644
---
a/server/src/test/java/org/apache/gravitino/server/web/rest/TestIcebergRESTServiceOperations.java
+++
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestIcebergRESTServiceOperations.java
@@ -20,6 +20,7 @@ package org.apache.gravitino.server.web.rest;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@@ -29,8 +30,11 @@ import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import org.apache.gravitino.auxiliary.AuxiliaryServiceManager;
+import org.apache.gravitino.dto.responses.ErrorResponse;
import org.apache.gravitino.dto.responses.IcebergRESTServiceResponse;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
public class TestIcebergRESTServiceOperations {
@@ -224,6 +228,93 @@ public class TestIcebergRESTServiceOperations {
assertEquals("http://irc-host:9001/iceberg";,
uriOf(ops.getIcebergRestServiceUri("")));
}
+ @Test
+ public void testAdvertisedUriIsReportedAsIs() {
+ // A reverse-proxied IRC: the public scheme, host, port and path all
differ from the listener's,
+ // and the path prefix and trailing slash must survive untouched.
+ IcebergRESTServiceOperations ops =
+ newOps(
+ true,
+ withDynamicProvider(
+ ImmutableMap.of(
+ "host", "0.0.0.0",
+ "httpPort", "9001",
+ "advertised-uri",
"https://iceberg.example.com:8443/proxy/iceberg/";)),
+ "gravitino.example.com");
+ assertEquals(
+ "https://iceberg.example.com:8443/proxy/iceberg/";,
uriOf(ops.getIcebergRestServiceUri("")));
+ }
+
+ @Test
+ public void testAdvertisedUriStillRequiresRegisteredAuxService() {
+ IcebergRESTServiceOperations ops =
+ newOps(
+ false,
+ withDynamicProvider(
+ ImmutableMap.of("advertised-uri",
"https://iceberg.example.com/iceberg";)),
+ "gravitino-host");
+ assertNull(uriOf(ops.getIcebergRestServiceUri("")));
+ }
+
+ @Test
+ public void testAdvertisedUriStillRequiresMatchingMetalake() {
+ IcebergRESTServiceOperations ops =
+ newOps(
+ true,
+ withDynamicProvider(
+ ImmutableMap.of(
+ "gravitino-metalake", "prod",
+ "advertised-uri", "https://iceberg.example.com/iceberg";)),
+ "gravitino-host");
+ assertNull(uriOf(ops.getIcebergRestServiceUri("test")));
+ assertEquals(
+ "https://iceberg.example.com/iceberg";,
uriOf(ops.getIcebergRestServiceUri("prod")));
+ }
+
+ @Test
+ public void testAdvertisedUriStillRequiresDynamicConfigProvider() {
+ IcebergRESTServiceOperations ops =
+ newOps(
+ true,
+ ImmutableMap.of("advertised-uri",
"https://iceberg.example.com/iceberg";),
+ "gravitino-host");
+ assertNull(uriOf(ops.getIcebergRestServiceUri("")));
+ }
+
+ @Test
+ public void testBlankAdvertisedUriFallsBackToDerivedEndpoint() {
+ IcebergRESTServiceOperations ops =
+ newOps(
+ true,
+ withDynamicProvider(ImmutableMap.of("host", "irc-host",
"advertised-uri", " ")),
+ "gravitino-host");
+ assertEquals("http://irc-host:9001/iceberg";,
uriOf(ops.getIcebergRestServiceUri("")));
+ }
+
+ @ParameterizedTest
+ @ValueSource(
+ strings = {
+ "iceberg.example.com/iceberg",
+ "ftp://iceberg.example.com/iceberg";,
+ "https:///iceberg";,
+ "https://iceberg.example.com/iceberg?x=1";,
+ "https://iceberg.example.com/iceberg#frag";,
+ "https://iceberg.example.com:0/iceberg";,
+ "https://iceberg.example.com:70000/iceberg";,
+ "http://bad host/iceberg"
+ })
+ public void testInvalidAdvertisedUriIsAnInternalError(String invalid) {
+ IcebergRESTServiceOperations ops =
+ newOps(
+ true,
+ withDynamicProvider(ImmutableMap.of("host", "irc-host",
"advertised-uri", invalid)),
+ "gravitino-host");
+ Response response = ops.getIcebergRestServiceUri("");
+ assertEquals(Response.Status.INTERNAL_SERVER_ERROR.getStatusCode(),
response.getStatus());
+ ErrorResponse error = (ErrorResponse) response.getEntity();
+ assertTrue(error.getMessage().contains("advertised-uri"));
+ }
+
@Test
public void testResponseIsNotCacheable() {
IcebergRESTServiceOperations ops =
