bhattmanish98 opened a new pull request, #8611: URL: https://github.com/apache/hadoop/pull/8611
### Description of PR Adds config-gated support for consuming **ListBlobs responses in the Apache Arrow (Photon) format** on the ABFS Blob endpoint, with automatic, transparent fallback to the existing XML path and **no public API changes**. JIRA: [HADOOP-19941](https://issues.apache.org/jira/browse/HADOOP-19941) ### Behaviour - New config `fs.azure.photon.enabled` (**default `false`**, opt-in). When enabled, ABFS advertises Arrow on ListBlobs via an `Accept` header (`application/vnd.apache.arrow.stream, application/xml`); the service may honour it or fall back to XML. - Response handling selects a parser by the response `Content-Type` — Arrow routes to the new Arrow parser, otherwise the existing XML SAX parser is used. Both produce the same `BlobListResultSchema`, so all downstream processing (rename-pending filtering, pagination, directory rectification) is unchanged. - Malformed Arrow responses surface as `AbfsDriverException` through the existing error-handling model rather than silently degrading. ### Implementation - **`ResponseParserFactory`** chooses between `XmlListBlobResponseParser` and `ArrowListBlobParser` behind the `ListBlobResponseParser` interface. - **`ArrowListBlobParser`** reaches full parity with the XML parser: blob user metadata (Metadata map column), `hdi_isfolder=true` directory markers (case-insensitive; empty `mkdir` directories), implicit directories (`BlobPrefix` / `ResourceType` `blobprefix`|`directory`), copy properties, native `TimeStampSec`/`UInt8` vectors, and continuation via schema `NextMarker` metadata. - Arrow/XML timestamps normalized to a single **RFC 1123 GMT** representation for identical `FileStatus` values; Arrow allocator memory limit is configurable. - **Interrupt-safe parsing**: reads the (already fully buffered) body through a non-interruptible channel instead of `ArrowStreamReader`'s interruptible NIO channel, avoiding `ClosedByInterruptException` on interrupted threads — matching the XML path. ### Telemetry Adds Photon metrics: request count, response (Arrow served) count, fallback (XML returned) count, parse-failure count, and an end-to-end listing-latency duration tracker. ### How was this patch tested? - **Unit tests**: parser selection, Arrow parsing (metadata/directory markers, blobprefix, native vector types, multi-page, special characters, allocator-limit, malformed streams, interrupt tolerance), request-header application, and metrics. - **Integration tests**: XML/Arrow parity, fallback, pagination, and metrics against a Blob-endpoint account, plus additions to the existing list-status ITest. - Documentation updated in `blobEndpoint.md`. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-19941. Your PR title ...')? - [x] Object storage: has the patch been tested against the target store (Azure Blob endpoint)? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF? (adds `org.apache.arrow:arrow-vector`, Apache-2.0) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
