aglinxinyuan opened a new issue, #7098: URL: https://github.com/apache/texera/issues/7098
### Task Summary `HDFSRecordStorage` is the only member of the record-storage family with no spec. Its three siblings in `amber/src/main/scala/org/apache/texera/amber/engine/common/storage/` are all covered (`EmptyRecordStorage`, `SequentialRecordStorage`, `VFSRecordStorage`), and the `hdfs://` arm of `SequentialRecordStorage.getStorage` is unexercised as well — `SequentialRecordStorageSpec` explicitly notes it was left out as "unit-test-hostile". It is testable, though, without a NameNode and without native Hadoop. The class builds its own `Configuration` internally so a test cannot inject one, but Hadoop's `FileSystem` cache is keyed on `(scheme, authority, UGI)` only — the `Configuration`'s contents are not part of the key. A test can therefore call `FileSystem.get` on an `hdfs://` URI with `fs.hdfs.impl` pointed at the repo's `WinutilsFreeLocalFileSystem` shim *before* constructing the class under test, and the internal `FileSystem.get` resolves to that same instance. A unique authority per test keeps cache entries isolated; the shim delegates to Hadoop's default local filesystem off Windows, so the same spec runs in CI. Branches worth covering: - the constructor's scheme guard, and its auto-create-folder branch - `getWriter` / `getReader` round-tripping records through the production serde - `getReader` on an absent file falling back to `EmptyRecordStorage` (an empty iterator, asserted as empty rather than assumed) - `deleteStorage` when the folder exists and when it does not - `containsFolder` for a directory vs a plain file vs something absent — the plain-file case is what an `exists`-only implementation would get wrong - `SequentialRecordStorage.getStorage` dispatching `hdfs://` and `HDFS://`, which is how production reaches the class The stale paragraph in `SequentialRecordStorageSpec` should be corrected at the same time: it claims regressions in the `hdfs` arm "would surface immediately in higher-level checkpoint / fault-tolerance suites that use hdfs:// URIs", but no suite in the repo uses an `hdfs://` URI. ### Task Type - [ ] Refactor / Cleanup - [ ] DevOps / Deployment / CI - [x] Testing / QA - [ ] Documentation - [ ] Performance - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
