sunchao commented on code in PR #5314:
URL: https://github.com/apache/datafusion-comet/pull/5314#discussion_r3876130764
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -42,6 +44,55 @@ object NativeConfig {
"abfs" -> Seq("fs.azure.", "fs.abfs."),
"abfss" -> Seq("fs.azure.", "fs.abfss.", "fs.abfs."))
+ private val blobKeyPattern = "^fs\\.blob\\.([^.]+)\\.(.+)$".r
+
+ // Some blob:// filesystem implementations fall back to the literal string
"default" as the
+ // authority when the URI has none. For `blob:///bucket/key`, the filesystem
therefore looks
+ // up `fs.blob.default.*`, while the actual S3 bucket comes from the URL
path. Translate
+ // `fs.blob.default.*` to the GLOBAL `fs.s3a.*` key (not the per-bucket
+ // `fs.s3a.bucket.default.*`) so the credentials/endpoint apply to whichever
bucket the URL
+ // path resolves to, matching those implementations' semantics.
+ private val blobDefaultAuthority = "default"
+
+ /**
+ * Translates vendor-style `fs.blob.<authority>.<property>` keys into the
`fs.s3a.*` shape that
+ * object_store's AmazonS3Builder reads. Some blob:// connectors use
per-authority keys and
+ * never set a region -- an endpoint alone is enough for the AWS SDK v1
client they build -- and
+ * their endpoints are typically path-style against non-AWS services, so an
`endpoint` key also
+ * enables `path.style.access` on the same scope.
+ *
+ * `fs.blob.<authority>.*` is the authoritative source for `blob://` URLs,
so callers should
+ * apply these translations AFTER a plain `fs.s3a.*` pass so blob-supplied
values override any
+ * unrelated `fs.s3a.*` the user set for a different workload (see 403
misdirect in the class
+ * docstring).
+ */
+ private def translateBlobKeys(hadoopConf: Configuration): Map[String,
String] = {
+ import scala.jdk.CollectionConverters._
+ val out = scala.collection.mutable.Map[String, String]()
+ hadoopConf.iterator().asScala.foreach { entry =>
+ entry.getKey match {
+ case blobKeyPattern(authority, property) =>
+ val s3aSuffix = property match {
+ case "endpoint" => Some("endpoint")
+ case "awsAccessKeyId" => Some("access.key")
+ case "awsSecretAccessKey" => Some("secret.key")
+ case _ => None
+ }
+ s3aSuffix.foreach { suffix =>
+ val scope =
+ if (authority == blobDefaultAuthority) "fs.s3a"
+ else s"fs.s3a.bucket.$authority"
Review Comment:
**[P2] Resolve authority-specific blob settings before constructing Iceberg
FileIO**
For `blob://mybucket/...` configured through `fs.blob.mybucket.*`, these
bucket-scoped keys become `s3.bucket.mybucket.*` in
`hadoopToIcebergS3Properties`. The pinned Iceberg S3 parser only consumes
global properties, so the endpoint, credentials and path style are ignored. A
local OpenDAL request probe selected the unrelated global endpoint/identity;
flattening the selected bucket properties fixed it. This parser limitation
already existed for s3a, but the new blob support now depends on it. Resolve
the relevant bucket settings to the properties FileIO actually consumes.
##########
spark/src/main/java/org/apache/comet/parquet/CometFileKeyUnwrapper.java:
##########
@@ -103,21 +103,27 @@ public class CometFileKeyUnwrapper {
/**
* Normalizes S3 URI schemes to a canonical form. S3 can be accessed via
multiple schemes (s3://,
- * s3a://, s3n://) that refer to the same logical filesystem. This method
ensures consistent cache
- * lookups regardless of which scheme is used.
+ * s3a://, s3n://, blob://) that refer to the same logical filesystem. This
method ensures
+ * consistent cache lookups regardless of which scheme is used. The put and
get sides must agree,
+ * because the JVM store side is called with the user-facing scheme (e.g.
blob://) while the
+ * native side JNIs back with the scheme after
`prepare_object_store_with_configs` has already
+ * rewritten aliases to s3://.
*
* @param filePath The file path that may contain an S3 URI
* @return The file path with normalized S3 scheme (s3a://)
*/
private String normalizeS3Scheme(final String filePath) {
- // Normalize s3:// and s3n:// to s3a:// for consistent cache lookups
- // This handles the case where ObjectStoreUrl uses s3:// but Spark uses
s3a://
+ // Normalize s3://, s3n://, and blob:// to s3a:// for consistent cache
lookups
+ // This handles the case where ObjectStoreUrl uses s3:// but Spark uses
s3a:// or blob://
String s3Prefix = "s3://";
String s3nPrefix = "s3n://";
+ String blobPrefix = "blob://";
if (filePath.startsWith(s3Prefix)) {
return "s3a://" + filePath.substring(s3Prefix.length());
} else if (filePath.startsWith(s3nPrefix)) {
return "s3a://" + filePath.substring(s3nPrefix.length());
+ } else if (filePath.startsWith(blobPrefix)) {
+ return "s3a://" + filePath.substring(blobPrefix.length());
Review Comment:
**[P2] Canonicalize single/triple-slash blob paths on both cache sides**
When Spark's input paths retain `blob:/bucket/key` or `blob:///bucket/key`,
native normalization promotes the bucket and calls back with `s3://bucket/key`.
This method leaves the single-slash path unchanged and maps the triple-slash
path to `s3a:///bucket/key`, so neither cache entry matches the callback's
`s3a://bucket/key`. Production Java cache methods with Spark 4.1.3 path
conversion reproduced `Failed to find DecryptionKeyRetriever` for both forms;
double-slash blob and s3a controls passed. Apply the same authority
normalization when storing and retrieving keys.
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -42,6 +44,55 @@ object NativeConfig {
"abfs" -> Seq("fs.azure.", "fs.abfs."),
"abfss" -> Seq("fs.azure.", "fs.abfss.", "fs.abfs."))
+ private val blobKeyPattern = "^fs\\.blob\\.([^.]+)\\.(.+)$".r
Review Comment:
**[P2] Keep dots inside blob authorities when translating settings**
For a valid dotted bucket such as `blob://my.bucket/data.parquet`,
`fs.blob.my.bucket.endpoint` is parsed as authority `my` and property
`bucket.endpoint`, which is discarded. The access-key and secret-key settings
are lost the same way. Compiling and calling the actual `NativeConfig` returned
an empty map for this case, while the `mybucket` control retained all settings.
Match the supported property suffixes and preserve the full authority,
otherwise native reads use missing or unrelated defaults.
##########
native/core/src/execution/planner.rs:
##########
@@ -4266,7 +4281,8 @@ fn parse_file_scan_tasks_from_common(
Ok(iceberg::scan::FileScanTask {
file_size_in_bytes: proto_task.file_size_in_bytes,
- data_file_path: proto_task.data_file_path.clone(),
+ // Normalize blob/s3a aliases so iceberg-rust routes through
S3, not LocalFs.
+ data_file_path:
normalize_object_store_url_string(&proto_task.data_file_path)?,
Review Comment:
**[P1] Preserve blob file identity for positional deletes**
Position-delete records retain the original `blob:` data-file path, but this
changes the task's identity to `s3:`. iceberg-rust then looks up a different
delete-vector key and silently returns deleted rows. With real Parquet and the
pinned ArrowReader over memory FileIO, `[10,20,30]` with position 1 deleted
returned `[10,30]` before normalization and `[10,20,30]` afterward for all
three blob slash forms; s3a/file controls passed. Preserve the original task
identity and normalize only storage access. This finding concerns positional
deletes, not equality deletes.
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -1054,6 +1054,26 @@ object CometScanRule extends Logging {
val SKIP_COMET_SCAN_TAG:
org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] =
org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometScan")
+ /**
+ * Schemes readable by iceberg-rust's OpenDAL storage factory that
`ObjectStoreScheme::parse`
+ * does NOT recognize, so `isNativelyReadableScheme` alone under-admits for
iceberg scans.
+ * Mirror of the extra `match` arms in
+ *
`native/core/src/execution/operators/iceberg_scan.rs::storage_factory_for` --
add here what
+ * you add there. Currently Aliyun OSS via `OpenDalStorageFactory::Oss`.
+ */
+ private val icebergExtraSchemes: Set[String] = Set("oss")
+
+ /**
+ * Scheme gate for the Iceberg scan path. Accepts anything the Parquet scan
gate accepts, plus
+ * schemes iceberg-rust reads via OpenDAL that object_store's parser doesn't
recognize.
+ */
+ private[rules] def isIcebergReadableScheme(uri: URI): Boolean = {
+ if (isNativelyReadableScheme(uri)) return true
Review Comment:
**[P2] Match Iceberg admission to its actual storage factory**
`object_store` accepts http/https, but `IcebergScanExec.storage_factory_for`
rejects those schemes. An S3FileIO table whose metadata/effective location and
data locations use `https://bucket/...` therefore changes from JVM fallback to
`Unsupported storage scheme: https`. Unmodified Iceberg Java S3FileIO and its
Parquet reader read `[10,20,30]` through this alias against a local S3
endpoint; the actual JNI gate accepted it and the exact native factory rejected
it. Keep the gate aligned with Iceberg's backends. HTTP data paths alone are
not the trigger when metadata remains s3.
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -42,6 +44,55 @@ object NativeConfig {
"abfs" -> Seq("fs.azure.", "fs.abfs."),
"abfss" -> Seq("fs.azure.", "fs.abfss.", "fs.abfs."))
+ private val blobKeyPattern = "^fs\\.blob\\.([^.]+)\\.(.+)$".r
+
+ // Some blob:// filesystem implementations fall back to the literal string
"default" as the
+ // authority when the URI has none. For `blob:///bucket/key`, the filesystem
therefore looks
+ // up `fs.blob.default.*`, while the actual S3 bucket comes from the URL
path. Translate
+ // `fs.blob.default.*` to the GLOBAL `fs.s3a.*` key (not the per-bucket
+ // `fs.s3a.bucket.default.*`) so the credentials/endpoint apply to whichever
bucket the URL
+ // path resolves to, matching those implementations' semantics.
+ private val blobDefaultAuthority = "default"
+
+ /**
+ * Translates vendor-style `fs.blob.<authority>.<property>` keys into the
`fs.s3a.*` shape that
+ * object_store's AmazonS3Builder reads. Some blob:// connectors use
per-authority keys and
+ * never set a region -- an endpoint alone is enough for the AWS SDK v1
client they build -- and
+ * their endpoints are typically path-style against non-AWS services, so an
`endpoint` key also
+ * enables `path.style.access` on the same scope.
+ *
+ * `fs.blob.<authority>.*` is the authoritative source for `blob://` URLs,
so callers should
+ * apply these translations AFTER a plain `fs.s3a.*` pass so blob-supplied
values override any
+ * unrelated `fs.s3a.*` the user set for a different workload (see 403
misdirect in the class
+ * docstring).
+ */
+ private def translateBlobKeys(hadoopConf: Configuration): Map[String,
String] = {
+ import scala.jdk.CollectionConverters._
+ val out = scala.collection.mutable.Map[String, String]()
+ hadoopConf.iterator().asScala.foreach { entry =>
+ entry.getKey match {
+ case blobKeyPattern(authority, property) =>
+ val s3aSuffix = property match {
+ case "endpoint" => Some("endpoint")
+ case "awsAccessKeyId" => Some("access.key")
+ case "awsSecretAccessKey" => Some("secret.key")
+ case _ => None
+ }
+ s3aSuffix.foreach { suffix =>
+ val scope =
+ if (authority == blobDefaultAuthority) "fs.s3a"
Review Comment:
**[P2] Preserve default-authority precedence after bucket promotion**
For `blob:///mybucket/...`, translating `fs.blob.default.*` only to global
`fs.s3a.*` does not override existing `fs.s3a.bucket.mybucket.*`: native
`get_config` checks the bucket scope first. An exact Scala/native configuration
probe selected an unrelated S3 endpoint and identity despite the configured
blob defaults. Translating a separate `fs.blob.mybucket.*` also overrides the
default authority for this authorityless URI. Resolve settings using the
original blob authority and apply them at the scope actually consulted after
normalization.
##########
native/core/src/execution/operators/iceberg_scan.rs:
##########
@@ -368,6 +368,22 @@ impl IcebergScanExec {
}
}
+ // Object-store's AmazonS3Builder defaults the SigV4 region to
`us-east-1` when unset;
+ // iceberg-storage-opendal's S3 factory instead errors with `region is
missing. Please
+ // find it by S3::detect_region() or set them in env.` Non-AWS
S3-compliant storage
+ // services accept any region in the credential, so default to
`us-east-1` when the
+ // catalog didn't ship one. Both key spellings iceberg-rust reads
(`client.region`
+ // wins over `s3.region`).
+ let region_forwarded = catalog_properties.contains_key("s3.region")
+ || catalog_properties.contains_key("client.region");
+ let is_s3_family = matches!(
+ metadata_location.split_once("://"),
+ Some(("s3" | "s3a" | "blob", _))
+ );
+ if !region_forwarded && is_s3_family {
+ file_io_builder = file_io_builder.with_prop("s3.region",
"us-east-1");
Review Comment:
**[P2] Honor the environment region before supplying a default**
When catalog properties omit a region, OpenDAL previously honored
`AWS_REGION`/`AWS_DEFAULT_REGION`. This unconditional property overrides that
behavior for existing s3/s3a tables too. A local HTTP probe using the exact
base/head `load_file_io` functions and dummy credentials signed for `us-west-2`
before this change and `us-east-1` afterward with `AWS_REGION=us-west-2`. AWS
buckets outside east-1 can therefore fail authentication. Honor the existing
region sources before defaulting, or restrict the fallback to the intended
non-AWS case.
##########
native/core/src/parquet/objectstore/blob_alias.rs:
##########
@@ -0,0 +1,235 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! S3-compatible filesystem alias handling.
+//!
+//! On the **Parquet** scan path Comet treats `blob://` and `s3a://` as
aliases for `s3://`: they
+//! route through the same `object_store::AmazonS3` store, but `object_store`'s
+//! `ObjectStoreScheme::parse` recognizes neither `blob` nor the `s3a`
dispatch Comet needs, so
+//! [`normalize_object_store_url`] rewrites both to canonical
`s3://bucket/key` (and promotes the
+//! single-slash `blob:/bucket/key` / three-slash `blob:///bucket/key` forms
to a real authority).
+//!
+//! On the **Iceberg** path the picture is narrower. iceberg-storage-opendal's
S3 backend is
+//! scheme-agnostic (it reads only `url.host_str()` for the bucket and strips
a `{scheme}://{bucket}/`
+//! prefix using the path's own scheme), so `s3a://` needs no rewrite there.
More importantly,
+//! iceberg-rust matches positional/equality deletes by comparing a delete
file's raw stored
+//! `file_path` against the `data_file_path` Comet supplies, so rewriting a
scheme silently drops
+//! deletes. [`normalize_object_store_url_string`] therefore rewrites ONLY the
s3-compliant alias
+//! schemes that genuinely require it ([`S3_COMPLIANT_ALIAS_SCHEMES`],
currently just `blob`) and
+//! returns every other input -- `file://`, `s3://`, `s3a://`, schemeless bare
paths -- byte-for-byte.
+//!
+//! The actual S3 store construction lives in [`super::s3`]; this module is
only about URL shape
+//! so all the alias code has one home instead of drifting across
`parquet_support.rs`,
+//! `execution/planner.rs`, and `execution/operators/iceberg_scan.rs`.
+
+use std::collections::HashMap;
+
+use url::Url;
+
+use crate::execution::operators::ExecutionError;
+use crate::parquet::parquet_support::is_hdfs_scheme;
+
+/// Parses `url_str` and rewrites `blob`/`s3a` schemes (and the awkward
three-slash
+/// `blob:///bucket/key` form that `ObjectStoreScheme::parse` rejects because
host=None) to the
+/// canonical `s3://bucket/key`. Non-alias schemes are returned unchanged.
+///
+/// `object_store_configs` is consulted only via `is_hdfs_scheme`: if the user
routed `s3a`
+/// through libhdfs via `fs.comet.libhdfs.schemes`, we must NOT rewrite it to
`s3` -- HDFS
+/// handling takes over.
+pub(crate) fn normalize_object_store_url(
+ url_str: &str,
+ object_store_configs: &HashMap<String, String>,
+) -> Result<Url, ExecutionError> {
+ let url = Url::parse(url_str)
+ .map_err(|e| ExecutionError::GeneralError(format!("Error parsing URL
{url_str}: {e}")))?;
+ if is_hdfs_scheme(&url, object_store_configs) {
+ return Ok(url);
+ }
+ // The Parquet scan path also rewrites `s3a` -> `s3` (pre-existing
behavior:
+ // `prepare_object_store_with_configs` dispatches on `scheme == "s3"`). It
only consumes
+ // `url.scheme()` / `url.path()` from the returned Url, never the
re-serialized string, so the
+ // `file:/x` -> `file:///x` round-trip that would break iceberg delete
matching is harmless
+ // here. See `normalize_object_store_url_string` for why the Iceberg path
is narrower.
+ let scheme = url.scheme();
+ if scheme != "s3a" && scheme != "blob" {
+ return Ok(url);
+ }
+ rewrite_alias_to_s3(url)
+}
+
+/// s3-compliant alias schemes Comet rewrites to canonical `s3://` on the
Iceberg path. Currently
+/// only `blob`. Kept deliberately narrow: iceberg-rust matches
positional/equality deletes by
+/// comparing the delete file's recorded `file_path` (raw, as Iceberg wrote
it) against the
+/// `data_file_path` Comet supplies (`delete_filter.rs::get_delete_vector`
looks up a map keyed by
+/// the delete file's `file_path` column in `caching_delete_file_loader.rs`).
Rewriting a scheme
+/// the delete files were NOT written with desyncs those two and silently
drops deletes, so only
+/// schemes that genuinely need an `s3://` shape to reach an object store
belong here.
+const S3_COMPLIANT_ALIAS_SCHEMES: &[&str] = &["blob"];
+
+/// Rewrites an already-parsed alias URL (`blob`/`s3a`) to canonical
`s3://bucket/key`, promoting
+/// the first path segment into the host for the single-slash
`blob:/bucket/key` (Java opaque
+/// form) and three-slash `blob:///bucket/key` (empty authority) shapes that
report host=None.
+fn rewrite_alias_to_s3(mut url: Url) -> Result<Url, ExecutionError> {
+ let original = url.scheme().to_string();
+ let needs_host_promotion = url.host_str().is_none();
+ url.set_scheme("s3").map_err(|_| {
+ ExecutionError::GeneralError(format!("Could not convert scheme from
{original} to s3"))
+ })?;
+ if needs_host_promotion {
+ // Some deployments emit `blob:///bucket/key` (three slashes, empty
authority) or Java
+ // collapses that to `blob:/bucket/key` (opaque form) in Iceberg
manifests. In both,
+ // `url::Url` reports host=None and path=`/bucket/key`, but
`ObjectStoreScheme::parse`
+ // requires a non-empty host. Lift the first path segment into the
host.
+ let trimmed = url.path().trim_start_matches('/').to_string();
+ let (bucket, key) = match trimmed.split_once('/') {
+ Some((b, k)) => (b.to_string(), k.to_string()),
+ None => (trimmed, String::new()),
+ };
+ if bucket.is_empty() {
+ return Err(ExecutionError::GeneralError(format!(
+ "{original}:// URL is missing bucket name: {url}"
+ )));
+ }
+ url = Url::parse(&format!("s3://{bucket}/{key}")).map_err(|e| {
+ ExecutionError::GeneralError(format!("Could not normalize
{original}:// URL: {e}"))
+ })?;
+ }
+ Ok(url)
+}
+
+/// String-returning normalizer for the Iceberg path, where paths are handed
to iceberg-rust as
+/// owned strings on `FileScanTask` / `FileScanTaskDeleteFile`.
+///
+/// Unlike the Url-returning [`normalize_object_store_url`], this MUST return
every non-`blob`
+/// input byte-for-byte. iceberg-rust keys its positional/equality delete maps
by the delete
+/// file's recorded `file_path` column (raw, as Iceberg Java wrote it) and
looks them up by the
+/// `data_file_path` Comet supplies here. Any rewrite Comet applies to
`data_file_path` that the
+/// delete file's stored path did not get -- `s3a://` -> `s3://`, or `url`'s
`file:/x` ->
+/// `file:///x` re-serialization -- desyncs the two and silently drops the
deletes (observed as a
+/// merge-on-read table returning rows that should have been deleted, and
rewrites failing to mark
+/// data files deleted).
+///
+/// So we only rewrite the s3-compliant alias schemes that genuinely need an
`s3://` shape to
+/// reach an object store ([`S3_COMPLIANT_ALIAS_SCHEMES`], currently just
`blob`). `file://`,
+/// `s3://`, `s3a://`, and schemeless bare paths (a local Hadoop-catalog
warehouse) all pass
+/// through unchanged; iceberg-storage-opendal's `storage_factory_for` routes
a `blob:/...` form
+/// correctly only after this rewrite and everything else via its own scheme
detection.
+pub(crate) fn normalize_object_store_url_string(path: &str) -> Result<String,
ExecutionError> {
+ let Ok(url) = Url::parse(path) else {
+ // Not a URL: a schemeless bare path from a local Hadoop-catalog
warehouse (e.g.
+ // `/tmp/warehouse/db/t/metadata/v1.metadata.json` from
`metadataFileLocation()`). Hand it
+ // to iceberg-storage-opendal's LocalFs backend unchanged, as the
raw-passthrough these
+ // call sites used before blob normalization did.
+ return Ok(path.to_string());
+ };
+ if !S3_COMPLIANT_ALIAS_SCHEMES.contains(&url.scheme()) {
+ // file://, s3://, s3a://, ... -> return the ORIGINAL string
byte-for-byte. Re-serializing
+ // through `url` (e.g. `String::from(url)`) rewrites the single-slash
`file:/x` to the
+ // three-slash `file:///x`, which no longer matches the delete file's
stored path.
+ return Ok(path.to_string());
+ }
+ // A recognized s3-compliant alias (blob) -> canonical `s3://bucket/key`,
promoting the bucket
+ // out of the path for the single/three-slash forms. `String::from(Url)`
moves the Url's
+ // serialization buffer out rather than re-serializing through `Display`.
+ Ok(String::from(rewrite_alias_to_s3(url)?))
Review Comment:
**[P2] Preserve raw Iceberg object keys during alias normalization**
A valid raw location such as `blob://bucket/warehouse/café/part.parquet`
becomes `s3://bucket/warehouse/caf%C3%A9/part.parquet` here. Iceberg's OpenDAL
adapter strips the prefix without decoding the key, and the S3 backend encodes
it again. A local HTTP capture showed `caf%25C3%25A9` in the request instead of
`caf%C3%A9`, so the scan targets a different object even without deletes.
Preserve the raw key while changing the scheme/authority; the unchanged s3a
path passed the same probe.
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -1095,12 +1112,14 @@ object CometScanRule extends Logging {
allParquet = false
}
- // Filesystem scheme check for data file
+ // Filesystem scheme check for data file. Delegated to the native gate
+ // (`isNativelyReadableScheme` ->
`NativeBase.isObjectStoreSchemeSupported`) so the iceberg
+ // and Parquet-scan paths share a single source of truth.
try {
val filePath = pathMethod.invoke(dataFile).toString
val uri = new URI(filePath)
val scheme = uri.getScheme
- if (scheme != null && !supportedSchemes.contains(scheme)) {
+ if (scheme != null && !isIcebergReadableScheme(uri)) {
Review Comment:
**[P2] Avoid caching raw Iceberg path failures as unsupported schemes**
Iceberg formats a newline partition value as the literal directory `p=%0A`,
which its native reader can read. Passing that raw path to `object_store`
decodes `%0A` into a control character and returns false. If this is the first
`file` check, `schemeSupportCache` stores false for the whole scheme, forcing
subsequent ordinary Iceberg and Parquet scans to fall back until JVM restart.
The actual reader returned `[10,20,30]` from this path; JNI accepted an
ordinary file URL, but the Scala gate rejected it after the first lookup. Use a
scheme-only probe rather than caching individual file-parse failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
