This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 07509a2dc67 [feature](fs-spi) add HTTP filesystem module, capability
framework and URI validation (#64853)
07509a2dc67 is described below
commit 07509a2dc67f12b9a2af318fb5a2a137a4af89c1
Author: Calvin Kirs <[email protected]>
AuthorDate: Wed Jul 8 21:08:04 2026 +0800
[feature](fs-spi) add HTTP filesystem module, capability framework and URI
validation (#64853)
### What problem does this PR solve?
Related PR: #xxx
Problem Summary:
An additive slice of the `fe-filesystem` SPI migration. It touches
**only `fe-filesystem`**
(zero `fe-core` footprint) and only lays down SPI geography that later
consumer-migration PRs
depend on:
1. **HTTP filesystem module** (`fe-filesystem-http`): typed
`HttpFileSystemProperties`
(validates `http://` / `https://` / `hf://` URIs, extracts
`http.header.*`) and
`HttpFileSystemProvider` registered via SPI. HTTP has no real FileSystem
— reads happen via the
`http()` TVF (FE `HttpUtils` / BE `TFileType.FILE_HTTP`) — so `create()`
throws; callers gate
via capabilities instead.
2. **Capability vocabulary**: `FileSystemCapability` enum
(READ/WRITE/LIST/DELETE/ATOMIC_RENAME/
HIERARCHICAL_NAMESPACE/APPEND/CONDITIONAL_WRITE) plus a config-aware
negotiation method on the
SPI, `FileSystemProvider.capabilities(P bound)` (with a
`capabilities(Map)` bridge that binds
first). Capability is negotiated against the resolved config, not the
provider type alone
(e.g. Ozone via S3 gateway has no ATOMIC_RENAME; via ofs:// it does). No
`fe-core` wiring /
selection shortcut is added here — that lands together with the first
real consumer, so no
speculative API ships without a caller.
3. **URI validation on the properties model**:
`FileSystemProperties.validateAndNormalizeUri()` /
`validateAndGetUri()` defaults. Pure config logic (no I/O); lives on the
properties model on
purpose — many callers validate a URI before, or without ever, creating
a FileSystem.
### Release note
None
### Check List (For Author)
- Test
- [x] Unit Test (`fe-filesystem-http`, `fe-filesystem-spi`,
`fe-filesystem-api` module tests)
- Behavior changed:
- [x] No. Additive SPI scaffolding; `fe-core` is untouched, no existing
path is rerouted.
- Does this need documentation?
- [x] No.
---
build.sh | 4 +-
.../properties/FileSystemCapability.java | 53 ++++++++++
.../properties/FileSystemProperties.java | 28 ++++++
fe/fe-filesystem/fe-filesystem-http/pom.xml | 74 ++++++++++++++
.../src/main/assembly/plugin-zip.xml | 65 ++++++++++++
.../filesystem/http/HttpFileSystemProperties.java | 109 +++++++++++++++++++++
.../filesystem/http/HttpFileSystemProvider.java | 73 ++++++++++++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 ++++
.../http/HttpFileSystemPropertiesTest.java | 90 +++++++++++++++++
.../http/HttpFileSystemProviderTest.java | 80 +++++++++++++++
.../doris/filesystem/spi/FileSystemProvider.java | 19 ++++
fe/fe-filesystem/pom.xml | 1 +
12 files changed, 612 insertions(+), 2 deletions(-)
diff --git a/build.sh b/build.sh
index e7dfb3d531c..7bcd0dc7439 100755
--- a/build.sh
+++ b/build.sh
@@ -717,7 +717,7 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
# Filesystem API and SPI plugin modules (loaded at runtime as plugins)
modules+=("fe-filesystem/fe-filesystem-api")
modules+=("fe-filesystem/fe-filesystem-spi")
- for _fs_mod in s3 oss cos obs azure hdfs local broker; do
+ for _fs_mod in s3 oss cos obs azure hdfs local broker http; do
if [[ -d "${DORIS_HOME}/fe/fe-filesystem/fe-filesystem-${_fs_mod}" ]];
then
modules+=("fe-filesystem/fe-filesystem-${_fs_mod}")
fi
@@ -1049,7 +1049,7 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
# Deploy filesystem provider plugins as independent plugin directories
# Each sub-directory is one storage backend loaded at runtime by
FileSystemPluginManager.
FS_PLUGIN_DIR="${DORIS_OUTPUT}/fe/plugins/filesystem"
- for fs_module in s3 azure oss cos obs hdfs local broker; do
+ for fs_module in s3 azure oss cos obs hdfs local broker http; do
fs_plugin_target="${FS_PLUGIN_DIR}/${fs_module}"
fs_module_dir="${DORIS_HOME}/fe/fe-filesystem/fe-filesystem-${fs_module}"
if [ ! -d "${fs_module_dir}" ]; then
diff --git
a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemCapability.java
b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemCapability.java
new file mode 100644
index 00000000000..f2080624550
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemCapability.java
@@ -0,0 +1,53 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.properties;
+
+/**
+ * Capabilities a filesystem provider may support.
+ *
+ * <p>Declared per provider (the storage type is the source of truth) so that
framework code can
+ * gate operations before invoking them, instead of calling and catching
UnsupportedOperationException.
+ */
+public enum FileSystemCapability {
+ /** Read an object. */
+ READ,
+ /** Write / upload an object. */
+ WRITE,
+ /** List / glob a prefix. */
+ LIST,
+ /** Delete an object. */
+ DELETE,
+ /**
+ * Rename is atomic and cheap. True filesystems (HDFS, local) provide
this; object stores
+ * emulate rename as copy+delete, which is neither atomic nor cheap, and
must not declare it.
+ */
+ ATOMIC_RENAME,
+ /**
+ * A real hierarchical namespace with directories (mkdirs /
renameDirectory / listDirectories
+ * carry meaning). Object stores only emulate this via key prefixes and
must not declare it.
+ */
+ HIERARCHICAL_NAMESPACE,
+ /** Append to an existing object. Supported by HDFS; not by object stores.
*/
+ APPEND,
+ /**
+ * Conditional write such as put-if-absent or if-match (used to build
commit locks). Supported
+ * by some object stores via conditional puts; true filesystems achieve
the same via
+ * {@link #ATOMIC_RENAME} instead.
+ */
+ CONDITIONAL_WRITE
+}
diff --git
a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemProperties.java
b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemProperties.java
index a0002ad6bf1..e75f5225129 100644
---
a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemProperties.java
+++
b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/FileSystemProperties.java
@@ -66,6 +66,34 @@ public interface FileSystemProperties extends
StorageProperties {
*/
Map<String, String> matchedProperties();
+ /**
+ * Validates and normalizes a single storage URI against this provider's
configuration.
+ *
+ * <p>This is pure configuration logic (no I/O) and lives on the
properties model rather than
+ * on {@link org.apache.doris.filesystem.FileSystem} on purpose: many
callers validate a URI
+ * before — or without ever — creating a FileSystem (e.g. HTTP has no
FileSystem). Providers
+ * with scheme/endpoint/path-style specific rules override this; the
default returns the URI
+ * unchanged.
+ *
+ * @throws IllegalArgumentException if the URI is invalid for this provider
+ */
+ default String validateAndNormalizeUri(String uri) {
+ return uri;
+ }
+
+ /**
+ * Extracts the storage URI from the given load properties and validates
it via
+ * {@link #validateAndNormalizeUri(String)}.
+ *
+ * <p>The default looks up the {@code "uri"} key. Providers whose URI
lives under a different
+ * key, or that need extra extraction logic (e.g. HDFS nameservices),
override this.
+ *
+ * @throws IllegalArgumentException if the URI is missing or invalid for
this provider
+ */
+ default String validateAndGetUri(Map<String, String> loadProperties) {
+ return validateAndNormalizeUri(loadProperties == null ? null :
loadProperties.get("uri"));
+ }
+
/**
* Converts to backend storage properties if this provider supports BE
access.
*/
diff --git a/fe/fe-filesystem/fe-filesystem-http/pom.xml
b/fe/fe-filesystem/fe-filesystem-http/pom.xml
new file mode 100644
index 00000000000..c71bc769f80
--- /dev/null
+++ b/fe/fe-filesystem/fe-filesystem-http/pom.xml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+ http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.doris</groupId>
+ <artifactId>fe-filesystem</artifactId>
+ <version>${revision}</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>fe-filesystem-http</artifactId>
+ <name>Doris FE Filesystem - HTTP</name>
+ <description>HTTP storage properties provider. Property binding only; HTTP
has no FileSystem
+ implementation (reads happen via the http() TVF on FE and
TFileType.FILE_HTTP on BE).</description>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.doris</groupId>
+ <artifactId>fe-filesystem-spi</artifactId>
+ <version>${revision}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <finalName>doris-fe-filesystem-http</finalName>
+ <plugins>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <configuration>
+ <appendAssemblyId>false</appendAssemblyId>
+ <descriptors>
+
<descriptor>src/main/assembly/plugin-zip.xml</descriptor>
+ </descriptors>
+ </configuration>
+ <executions>
+ <execution>
+ <id>make-assembly</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/main/assembly/plugin-zip.xml
b/fe/fe-filesystem/fe-filesystem-http/src/main/assembly/plugin-zip.xml
new file mode 100644
index 00000000000..e65d143ff9b
--- /dev/null
+++ b/fe/fe-filesystem/fe-filesystem-http/src/main/assembly/plugin-zip.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!--
+ Plugin zip layout expected by DirectoryPluginRuntimeManager:
+ <finalName>.jar (the plugin jar at root — discovered for service
registration)
+ lib/
+ *.jar (all runtime deps: direct + transitive)
+
+ The plugin jar is placed at root via <files> so that service-discovery scans
+ only root-level jars. All dependencies (including intra-doris modules such as
+ fe-filesystem-s3 inside fe-filesystem-cos) land in lib/ and are available for
+ class loading but are NOT scanned for FileSystemProvider registrations.
+
+ Jars already present in fe-core classpath are excluded from lib/:
+ fe-filesystem-api, fe-filesystem-spi, fe-extension-spi
+-->
+<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.2.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0
+ https://maven.apache.org/xsd/assembly-2.2.0.xsd";>
+ <id>plugin</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <!-- Plugin jar at the root of the zip (service-discovery target) -->
+ <files>
+ <file>
+
<source>${project.build.directory}/${project.build.finalName}.jar</source>
+ <outputDirectory>/</outputDirectory>
+ </file>
+ </files>
+
+ <dependencySets>
+ <!-- All runtime deps (direct + transitive) in lib/ -->
+ <dependencySet>
+ <outputDirectory>/lib</outputDirectory>
+ <useProjectArtifact>false</useProjectArtifact>
+ <scope>runtime</scope>
+ <excludes>
+ <exclude>org.apache.doris:fe-filesystem-api</exclude>
+ <exclude>org.apache.doris:fe-filesystem-spi</exclude>
+ <exclude>org.apache.doris:fe-extension-spi</exclude>
+ </excludes>
+ </dependencySet>
+ </dependencySets>
+</assembly>
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProperties.java
b/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProperties.java
new file mode 100644
index 00000000000..be13901f0dc
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProperties.java
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.http;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageKind;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * Typed properties for HTTP storage. HTTP is not a real filesystem: reads
happen via the
+ * http() TVF on FE (HttpUtils) and TFileType.FILE_HTTP on BE. This class only
validates the URI,
+ * extracts http.header.* headers, and passes the raw map through for BE
backend config.
+ */
+public class HttpFileSystemProperties implements FileSystemProperties {
+
+ public static final String URI_KEY = "uri";
+ public static final String HEADER_PREFIX = "http.header.";
+
+ private final Map<String, String> rawProperties;
+ private final String uri;
+ private final Map<String, String> headers;
+ private final Map<String, String> matchedProperties;
+
+ private HttpFileSystemProperties(Map<String, String> rawProperties) {
+ this.rawProperties = Collections.unmodifiableMap(new
HashMap<>(rawProperties));
+ this.uri = validateUri(rawProperties.get(URI_KEY));
+ Map<String, String> hdrs = new LinkedHashMap<>();
+ Map<String, String> matched = new LinkedHashMap<>();
+ matched.put(URI_KEY, this.uri);
+ for (Map.Entry<String, String> e : rawProperties.entrySet()) {
+ if (e.getKey().toLowerCase().startsWith(HEADER_PREFIX)) {
+ hdrs.put(e.getKey().substring(HEADER_PREFIX.length()),
e.getValue());
+ matched.put(e.getKey(), e.getValue());
+ }
+ }
+ this.headers = Collections.unmodifiableMap(hdrs);
+ this.matchedProperties = Collections.unmodifiableMap(matched);
+ }
+
+ public static HttpFileSystemProperties of(Map<String, String> properties) {
+ return new HttpFileSystemProperties(properties);
+ }
+
+ private static String validateUri(String url) {
+ if (url == null
+ || (!url.startsWith("http://";) && !url.startsWith("https://";)
&& !url.startsWith("hf://"))) {
+ throw new IllegalArgumentException("Invalid http/hf url: " + url);
+ }
+ return url;
+ }
+
+ @Override
+ public String validateAndNormalizeUri(String url) {
+ return validateUri(url);
+ }
+
+ public String getUri() {
+ return uri;
+ }
+
+ public Map<String, String> getHeaders() {
+ return headers;
+ }
+
+ @Override
+ public String providerName() {
+ return "HTTP";
+ }
+
+ @Override
+ public StorageKind kind() {
+ return StorageKind.HTTP;
+ }
+
+ @Override
+ public FileSystemType type() {
+ return FileSystemType.HTTP;
+ }
+
+ @Override
+ public Map<String, String> rawProperties() {
+ return rawProperties;
+ }
+
+ @Override
+ public Map<String, String> matchedProperties() {
+ return matchedProperties;
+ }
+}
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProvider.java
b/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProvider.java
new file mode 100644
index 00000000000..0ebeb002e60
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-http/src/main/java/org/apache/doris/filesystem/http/HttpFileSystemProvider.java
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.http;
+
+import org.apache.doris.filesystem.FileSystem;
+import org.apache.doris.filesystem.properties.FileSystemCapability;
+import org.apache.doris.filesystem.spi.FileSystemProvider;
+
+import java.util.EnumSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * SPI provider for HTTP storage (http://, https://, hf:// URIs).
+ *
+ * <p>HTTP has no FileSystem implementation: the http() TVF reads via
HttpUtils on FE and
+ * TFileType.FILE_HTTP on BE. This provider only binds and validates
properties; {@link #create}
+ * throws. Use {@link #capabilities(HttpFileSystemProperties)} (READ) to gate
operations
+ * instead of calling create().
+ */
+public class HttpFileSystemProvider implements
FileSystemProvider<HttpFileSystemProperties> {
+
+ @Override
+ public boolean supports(Map<String, String> properties) {
+ if ("HTTP".equalsIgnoreCase(properties.get("_STORAGE_TYPE_"))) {
+ return true;
+ }
+ String uri = properties.getOrDefault("uri", "");
+ return uri.startsWith("http://";) || uri.startsWith("https://";) ||
uri.startsWith("hf://");
+ }
+
+ @Override
+ public HttpFileSystemProperties bind(Map<String, String> properties) {
+ return HttpFileSystemProperties.of(properties);
+ }
+
+ @Override
+ public FileSystem create(HttpFileSystemProperties properties) {
+ throw new UnsupportedOperationException(
+ "HTTP has no FileSystem; read via the http() TVF (FE HttpUtils
/ BE FILE_HTTP).");
+ }
+
+ @Override
+ public FileSystem create(Map<String, String> properties) {
+ throw new UnsupportedOperationException(
+ "HTTP has no FileSystem; read via the http() TVF (FE HttpUtils
/ BE FILE_HTTP).");
+ }
+
+ @Override
+ public Set<FileSystemCapability> capabilities(HttpFileSystemProperties
boundProperties) {
+ return EnumSet.of(FileSystemCapability.READ);
+ }
+
+ @Override
+ public String name() {
+ return "HTTP";
+ }
+}
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
b/fe/fe-filesystem/fe-filesystem-http/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
new file mode 100644
index 00000000000..239cf71ddc9
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-http/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+org.apache.doris.filesystem.http.HttpFileSystemProvider
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemPropertiesTest.java
new file mode 100644
index 00000000000..b9f68f46343
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemPropertiesTest.java
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.http;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.StorageKind;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class HttpFileSystemPropertiesTest {
+
+ private Map<String, String> props(String uri) {
+ Map<String, String> m = new HashMap<>();
+ m.put("uri", uri);
+ return m;
+ }
+
+ @Test
+ public void testAcceptsHttpHttpsHfSchemes() {
+ Assertions.assertEquals("http://h/a.csv";,
HttpFileSystemProperties.of(props("http://h/a.csv";)).getUri());
+ Assertions.assertEquals("https://h/a.csv";,
HttpFileSystemProperties.of(props("https://h/a.csv";)).getUri());
+ Assertions.assertEquals("hf://ds/a",
HttpFileSystemProperties.of(props("hf://ds/a")).getUri());
+ }
+
+ @Test
+ public void testRejectsOtherSchemes() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HttpFileSystemProperties.of(props("s3://b/k")));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HttpFileSystemProperties.of(props(null)));
+ }
+
+ @Test
+ public void testExtractsHeaders() {
+ Map<String, String> m = props("http://h/a.csv";);
+ m.put("http.header.Authorization", "Bearer x");
+ m.put("http.header.Accept", "text/csv");
+ m.put("format", "csv");
+ Map<String, String> headers =
HttpFileSystemProperties.of(m).getHeaders();
+ Assertions.assertEquals(2, headers.size());
+ Assertions.assertEquals("Bearer x", headers.get("Authorization"));
+ Assertions.assertEquals("text/csv", headers.get("Accept"));
+ }
+
+ @Test
+ public void testValidateAndNormalizeUri() {
+ HttpFileSystemProperties p =
HttpFileSystemProperties.of(props("http://h/a.csv";));
+ Assertions.assertEquals("https://h/b.csv";,
p.validateAndNormalizeUri("https://h/b.csv";));
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
p.validateAndNormalizeUri("s3://b/k"));
+ }
+
+ @Test
+ public void testValidateAndGetUri() {
+ HttpFileSystemProperties p =
HttpFileSystemProperties.of(props("http://h/a.csv";));
+ Map<String, String> load = new HashMap<>();
+ load.put("uri", "hf://ds/x");
+ Assertions.assertEquals("hf://ds/x", p.validateAndGetUri(load));
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
p.validateAndGetUri(new HashMap<>()));
+ }
+
+ @Test
+ public void testTypeAndKindAndRaw() {
+ Map<String, String> m = props("http://h/a.csv";);
+ HttpFileSystemProperties p = HttpFileSystemProperties.of(m);
+ Assertions.assertEquals("HTTP", p.providerName());
+ Assertions.assertEquals(FileSystemType.HTTP, p.type());
+ Assertions.assertEquals(StorageKind.HTTP, p.kind());
+ Assertions.assertEquals("http://h/a.csv";,
p.rawProperties().get("uri"));
+ Assertions.assertTrue(p.matchedProperties().containsKey("uri"));
+ }
+}
diff --git
a/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemProviderTest.java
b/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemProviderTest.java
new file mode 100644
index 00000000000..6d6f6b311ed
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-http/src/test/java/org/apache/doris/filesystem/http/HttpFileSystemProviderTest.java
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.http;
+
+import org.apache.doris.filesystem.properties.FileSystemCapability;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.Map;
+
+public class HttpFileSystemProviderTest {
+
+ private final HttpFileSystemProvider provider = new
HttpFileSystemProvider();
+
+ private Map<String, String> props(String key, String value) {
+ Map<String, String> m = new HashMap<>();
+ m.put(key, value);
+ return m;
+ }
+
+ @Test
+ public void testSupportsByScheme() {
+ Assertions.assertTrue(provider.supports(props("uri",
"http://h/a.csv";)));
+ Assertions.assertTrue(provider.supports(props("uri",
"https://h/a.csv";)));
+ Assertions.assertTrue(provider.supports(props("uri", "hf://ds/a")));
+ Assertions.assertFalse(provider.supports(props("uri", "s3://b/k")));
+ Assertions.assertFalse(provider.supports(new HashMap<>()));
+ }
+
+ @Test
+ public void testSupportsByMarker() {
+ Assertions.assertTrue(provider.supports(props("_STORAGE_TYPE_",
"HTTP")));
+ }
+
+ @Test
+ public void testBindReturnsHttpProperties() {
+ Map<String, String> m = props("uri", "http://h/a.csv";);
+ m.put("http.header.Accept", "text/csv");
+ FileSystemProperties fp = provider.bind(m);
+ Assertions.assertTrue(fp instanceof HttpFileSystemProperties);
+ Assertions.assertEquals("http://h/a.csv";, ((HttpFileSystemProperties)
fp).getUri());
+ Assertions.assertEquals("text/csv", ((HttpFileSystemProperties)
fp).getHeaders().get("Accept"));
+ }
+
+ @Test
+ public void testCreateThrows() {
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () -> provider.create(props("uri", "http://h/a.csv";)));
+ }
+
+ @Test
+ public void testCapabilitiesReadOnly() {
+ HttpFileSystemProperties bound = provider.bind(props("uri",
"http://h/a.csv";));
+ Assertions.assertEquals(EnumSet.of(FileSystemCapability.READ),
provider.capabilities(bound));
+ }
+
+ @Test
+ public void testName() {
+ Assertions.assertEquals("HTTP", provider.name());
+ }
+}
diff --git
a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java
b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java
index a44a005ed61..d6bde78fce3 100644
---
a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java
+++
b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java
@@ -20,10 +20,12 @@ package org.apache.doris.filesystem.spi;
import org.apache.doris.extension.spi.Plugin;
import org.apache.doris.extension.spi.PluginFactory;
import org.apache.doris.filesystem.FileSystem;
+import org.apache.doris.filesystem.properties.FileSystemCapability;
import org.apache.doris.filesystem.properties.FileSystemProperties;
import java.io.IOException;
import java.util.Collections;
+import java.util.EnumSet;
import java.util.Map;
import java.util.Set;
@@ -109,6 +111,23 @@ public interface FileSystemProvider<P extends
FileSystemProperties> extends Plug
return Collections.emptySet();
}
+ /**
+ * Negotiates the capabilities this provider exposes for the given bound
configuration.
+ *
+ * <p>Capability is a function of the resolved configuration, not of the
provider type alone:
+ * the same provider may expose different capabilities depending on the
config (e.g. Ozone via
+ * the S3 gateway has no {@link FileSystemCapability#ATOMIC_RENAME}, but
Ozone via {@code ofs://}
+ * does). Defaults to the empty set; providers override to declare what
they support.
+ *
+ * <p>Capability negotiation is intentionally typed: the caller binds the
raw property map via
+ * {@link #bind(Map)} first, then negotiates against the resulting
configuration. There is no
+ * raw-map bridge here — a legacy provider that has not migrated to {@link
#bind(Map)} cannot be
+ * negotiated against, and a silent fallback would hide that instead of
surfacing it.
+ */
+ default Set<FileSystemCapability> capabilities(P boundProperties) {
+ return EnumSet.noneOf(FileSystemCapability.class);
+ }
+
/**
* Human-readable name for logging/diagnostics (e.g., "S3", "HDFS",
"Azure").
*/
diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml
index e49b52aaf8a..7333615dbd5 100644
--- a/fe/fe-filesystem/pom.xml
+++ b/fe/fe-filesystem/pom.xml
@@ -54,6 +54,7 @@ under the License.
<module>fe-filesystem-azure</module>
<module>fe-filesystem-hdfs</module>
<module>fe-filesystem-broker</module>
+ <module>fe-filesystem-http</module>
</modules>
<build>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
