comphead opened a new pull request, #7157: URL: https://github.com/apache/datafusion-comet/pull/7157
## Which issue does this PR close? Closes #5894. ## Rationale for this change apache/iceberg-rust#3111, merged as `ca21f158`, adds a Rust HDFS backend to `iceberg-storage-opendal` behind the `opendal-hdfs-native` feature. It uses OpenDAL's `services-hdfs-native`, which needs neither JNI nor libhdfs. Comet's native Iceberg scan has no `hdfs` arm, so every Iceberg table on HDFS falls back to the JVM reader. This builds on @mixermt's design in #5898, which was written against an earlier revision of apache/iceberg-rust#3111. The merged backend changed two things that shape this PR: - A plain `hdfs.name-node` no longer overrides a path's authority. It serves only authority-less paths, and the backend caches one operator per effective NameNode. #5898's single-NameNode-per-scan gate is therefore not needed. - A portless authority such as `hdfs://ns1/...` must be a nameservice declared in the FileIO properties, as `hdfs.name-node.ns1` or with Hadoop's own keys under `hadoop.`. The backend never resolves it from `$HADOOP_CONF_DIR`. ## What changes are included in this PR? - Moves the iceberg-rust pin from `1f3bc340` to `ca21f158` (52 commits) and enables `opendal-hdfs-native`. `hdfs-native` resolves to 0.14.7. Versions before 0.14.5 fail every read on a cluster with `dfs.checksum.type=CRC32` (Kimahriman/hdfs-native#323). - Native: `storage_factory_for` routes `hdfs` to `OpenDalStorageFactory::HdfsNative`, `hdfs` joins the read list in `builtin_storage_schemes`, and `hdfs.` and `hadoop.` join `STORAGE_PROPERTY_PREFIXES`. Writes stay out of the list. - JVM: `validateIcebergFileScanTasks` collects the HDFS authorities of the data and delete files. `hadoopToIcebergHdfsProperties` declares each portless one as `hdfs.name-node.<ns>`, from the session's `dfs.ha.namenodes.<ns>` and `dfs.namenode.rpc-address.<ns>.<id>`, with the NameNodes in declaration order. It is merged before the catalog's properties, so a catalog's own declaration wins. The scan falls back when a portless authority stays undeclared, instead of failing at execution. - JVM: the scan falls back when it reads HDFS and `hadoop.security.authentication` is not `simple`. OpenDAL's hdfs-native service builds every client for the synthetic nameservice `nameservice` (`init_hdfs_config` in `opendal-service-hdfs-native` 0.58.2). hdfs-native picks a delegation token by `ha-hdfs:<nameservice>` (`RpcConnection::connect`, 0.14.7). A real token's service is `ha-hdfs:<ns>` or a NameNode address, so an executor, which holds tokens and no TGT, would fail to authenticate. With a print added to `RpcConnection::connect`, a FileIO built as this PR builds it looks up `ha-hdfs:nameservice` for both `hdfs://ns1/...` and `hdfs://host:port/...`, while hdfs-native on its own, given `hdfs://ns1`, looks up `ha-hdfs:ns1`. The Parquet scan is not affected: it reaches HDFS through OpenDAL's `services-hdfs`, which is libhdfs over JNI and uses the executor JVM's credentials. - Docs: an HDFS section in the Iceberg guide, and the gaps below. ## How are these changes tested? - Rust (`iceberg_common`): `hdfs` resolves for reads and is rejected for writes. `hdfs.` and `hadoop.` properties reach the FileIO. The tests that used `hdfs` as the example of an unsupported scheme now use `viewfs`. - JVM: `CometIcebergNativeScanSuite` covers the nameservice declaration: declaration order, a NameNode without a port, an undeclared nameservice, an authority with a port, and a catalog's own declaration. `CometScanSchemeFallbackSuite` checks that `hdfs` is in the read list and not in the write list, and that a hostless `hdfs:///` location is not openable. - There is no end-to-end test against an HDFS cluster. `WithHdfsCluster` is used only by `CometReadBenchmark`. #5898 reports that the 3.3.4 MiniDFSCluster does not start on the Spark 4.x profiles. - Checked locally: `cargo clippy --all-targets --workspace -- -D warnings`, `cargo fmt --check`, `./mvnw test-compile` (Spark 4.1, with scalastyle), the strict warnings build on `-Pspark-3.5`, and `prettier --check` on the changed docs. The tests themselves have not run yet. ### Depends on #7028 #7028 moves the same pin to `b154603a`. Both moves include apache/iceberg-rust#3255. After #3255, a pruned read of a struct in a data file with neither field ids nor a name mapping fails with "cannot reconcile struct fields by id". #7028 updates the `CometIcebergNativeSuite` test that covers this, and this branch does not, so it should be rebased after #7028 merges. The pin move also brings 40 more iceberg-rust commits than #7028, several of them in the reader, so this needs `run-iceberg-tests`. ### Not covered here - Kerberos clusters fall back, as described above. Fixing that needs OpenDAL to build the client for the real nameservice, or hdfs-native to find tokens by NameNode address. - On a cluster without Kerberos, the client connects as `HADOOP_USER_NAME`, or as the executor's operating system user, rather than as the Spark user. - Reads of files in an HDFS encryption zone fail, because OpenDAL does not enable hdfs-native's `kms` feature. - Writes to HDFS keep Iceberg's writer. - iceberg-rust runs HDFS clients on its own multi-thread Tokio runtime, `iceberg-hdfs-native`, one per process. - The FileIO cache keys on the metadata location, so each new snapshot of a table builds a new HDFS client and new NameNode connections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
