jordepic commented on code in PR #2441:
URL: https://github.com/apache/iceberg-rust/pull/2441#discussion_r3282264987
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -218,6 +234,20 @@ pub enum OpenDalStorage {
/// Azure DLS configuration.
config: Arc<AzdlsConfig>,
},
+ /// HDFS storage variant (via OpenDAL `services-hdfs-native`).
+ ///
+ /// Accepts paths of the form `hdfs://<authority>/<rel>` (or
+ /// `hdfs:///<rel>` for authority-less paths, which defer to
+ /// `fs.defaultFS` from the `hdfs-native`-loaded Hadoop config). The
+ /// authority - or `None` when absent - keys a per-name-node `Operator`
+ /// cache, so distinct name nodes coexist in a single storage instance.
+ #[cfg(feature = "opendal-hdfs-native")]
+ Hdfs {
+ /// Operator cache. `Some("hdfs://<authority>")` for paths with an
+ /// authority; `None` for authority-less paths (`fs.defaultFS`).
+ #[serde(skip, default)]
+ operators: Arc<RwLock<HashMap<Option<String>, Operator>>>,
Review Comment:
Considered it but the cache is by nature unbounded-friendly and tiny: one
entry per HDFS name service, so typically 1–3 entries (router-based federation
might push it to ~10). The properties moka brings
— LRU/LFU eviction, TTL, bounded size — don't apply, and the lock-free read
path doesn't help much when our RwLock readers hold for sub-microseconds.
Staying on stdlib also keeps the workspace dep set
minimal. Happy to switch if you feel strongly about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
