[
https://issues.apache.org/jira/browse/HADOOP-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096747#comment-18096747
]
ASF GitHub Bot commented on HADOOP-19941:
-----------------------------------------
bhattmanish98 opened a new pull request, #8611:
URL: https://github.com/apache/hadoop/pull/8611
### Description of PR
Adds config-gated support for consuming **ListBlobs responses in the Apache
Arrow (Photon) format** on the ABFS Blob endpoint, with automatic, transparent
fallback to the existing XML path and **no public API changes**.
JIRA: [HADOOP-19941](https://issues.apache.org/jira/browse/HADOOP-19941)
### Behaviour
- New config `fs.azure.photon.enabled` (**default `false`**, opt-in). When
enabled, ABFS advertises Arrow on ListBlobs via an `Accept` header
(`application/vnd.apache.arrow.stream, application/xml`); the service may
honour it or fall back to XML.
- Response handling selects a parser by the response `Content-Type` — Arrow
routes to the new Arrow parser, otherwise the existing XML SAX parser is used.
Both produce the same `BlobListResultSchema`, so all downstream processing
(rename-pending filtering, pagination, directory rectification) is unchanged.
- Malformed Arrow responses surface as `AbfsDriverException` through the
existing error-handling model rather than silently degrading.
### Implementation
- **`ResponseParserFactory`** chooses between `XmlListBlobResponseParser`
and `ArrowListBlobParser` behind the `ListBlobResponseParser` interface.
- **`ArrowListBlobParser`** reaches full parity with the XML parser: blob
user metadata (Metadata map column), `hdi_isfolder=true` directory markers
(case-insensitive; empty `mkdir` directories), implicit directories
(`BlobPrefix` / `ResourceType` `blobprefix`|`directory`), copy properties,
native `TimeStampSec`/`UInt8` vectors, and continuation via schema `NextMarker`
metadata.
- Arrow/XML timestamps normalized to a single **RFC 1123 GMT**
representation for identical `FileStatus` values; Arrow allocator memory limit
is configurable.
- **Interrupt-safe parsing**: reads the (already fully buffered) body
through a non-interruptible channel instead of `ArrowStreamReader`'s
interruptible NIO channel, avoiding `ClosedByInterruptException` on interrupted
threads — matching the XML path.
### Telemetry
Adds Photon metrics: request count, response (Arrow served) count, fallback
(XML returned) count, parse-failure count, and an end-to-end listing-latency
duration tracker.
### How was this patch tested?
- **Unit tests**: parser selection, Arrow parsing (metadata/directory
markers, blobprefix, native vector types, multi-page, special characters,
allocator-limit, malformed streams, interrupt tolerance), request-header
application, and metrics.
- **Integration tests**: XML/Arrow parity, fallback, pagination, and metrics
against a Blob-endpoint account, plus additions to the existing list-status
ITest.
- Documentation updated in `blobEndpoint.md`.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-19941. Your PR title ...')?
- [x] Object storage: has the patch been tested against the target store
(Azure Blob endpoint)?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under ASF? (adds
`org.apache.arrow:arrow-vector`, Apache-2.0)
> ABFS: Support Photon (Apache Arrow) based ListBlobs on Blob endpoint with XML
> fallback
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-19941
> URL: https://issues.apache.org/jira/browse/HADOOP-19941
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
> Reporter: Manish Bhatt
> Assignee: Manish Bhatt
> Priority: Major
>
> Add config-gated support (fs.azure.photon.enabled, default off) for consuming
> ListBlobs responses in the Apache Arrow (Photon) format on the ABFS Blob
> endpoint. When enabled, ABFS advertises Arrow via an Accept header; the
> response Content-Type selects an Arrow or the existing XML parser, both
> producing identical FileStatus results, so downstream behaviour is unchanged.
> Includes automatic, transparent fallback to XML, full parsing parity
> (metadata, directory markers, implicit directories, copy properties, native
> timestamp/length vectors), interrupt-safe Arrow parsing, and Photon telemetry
> (request, response, fallback, parse-failure counts and listing latency). No
> public API changes. Covered by unit and integration tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
