comphead opened a new issue, #5313:
URL: https://github.com/apache/datafusion-comet/issues/5313
### What is the problem the feature request solves?
### Motivation
Comet today accepts only `s3://` and `s3a://` URIs, which effectively
couples native execution to AWS S3 and the Hadoop `s3a` connector. In practice,
most production Spark deployments live on S3-compliant
storage that speaks the S3 protocol against a non-AWS endpoint. Users on
these systems either lose Comet acceleration entirely or have to rewrite their
table locations to fit the `s3a` mold.
Common S3-compliant systems Spark users route through today:
- **Google Cloud Storage** via its S3-compatible interoperability endpoint
(`storage.googleapis.com`).
- **Cloudflare R2**, **Backblaze B2**, **Wasabi**, **DigitalOcean
Spaces**, **Linode Object Storage**.
- **MinIO** (self-hosted and as the storage layer for many Kubernetes /
on-prem data platforms).
- **Ceph RADOS Gateway**, **SeaweedFS**, **OpenStack Swift** (with the S3
middleware).
- **IBM Cloud Object Storage**, **Oracle Cloud Object Storage** (S3
compatibility API).
- **Dell ECS**, **NetApp StorageGRID**, **Pure FlashBlade** and other
enterprise on-prem object stores.
- Internal / vendor-specific S3-compliant blob services that many
organizations expose through a custom `blob://` (or similarly aliased) Hadoop
`FileSystem`.
These systems share a common shape: S3 protocol, custom endpoint, usually
path-style addressing, frequently no meaningful region, and often a
vendor-specific config namespace instead of `fs.s3a.*`. The
current gate treats any of these as "unsupported filesystem" and falls
back to Spark, which defeats the purpose of Comet on precisely the deployments
that would benefit most.
### Goal
Allow Comet to natively read (and Iceberg-scan) tables backed by any
S3-compliant object store, regardless of the URI scheme the user's Hadoop
`FileSystem` exposes or which vendor namespace they configure
credentials under. Behavior on AWS S3 and the standard `s3a` connector
must remain unchanged.
### Out of scope
- Native writes to S3-compliant stores (there is no S3 write path today).
- Non-S3 protocols (GCS native HMAC / OAuth, Azure Blob / ADLS Gen2).
Those already have their own paths and are not affected.
### Compatibility
Additive. Users on AWS S3 or `s3a` see no change. Users on S3-compliant
storage go from "falls back to Spark" to "runs natively in Comet".
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
