bhattmanish98 opened a new pull request, #8830: URL: https://github.com/apache/hadoop/pull/8830
### Description of PR Introduces blob-layout-based data locality for the ABFS driver's Blob endpoint client, allowing `AbfsInputStream` reads to fetch data directly from the storage node hosting the relevant byte range rather than a generic endpoint. Adds the `BlobLayout` model, an XML parser for the layout response, a process-wide `BlobLayoutCache` with weight-based eviction and non-blocking async prefetching backed by a dedicated executor, and wiring in `AbfsBlobClient`/`AbfsInputStream` to look up and use endpoint-aware reads with correctness fallback (partial/corrupted/ evicted-cache scenarios) to the standard read path. The feature is gated behind the `fs.azure.enable.data.locality` configuration flag, which defaults to `false` (disabled), so existing behavior is unchanged unless explicitly opted in. ### How was this patch tested? Includes unit and integration tests covering `BlobLayout` parsing, caching, eviction (capacity/idle/weight-based), and `AbfsInputStream` read behavior across scenarios: data not returned, corrupted/partial responses, and cache eviction during read. Additionally manually validated end-to-end against a live Azure Storage account with the Blob endpoint and data locality enabled, confirming the blob-layout lookup, caching, and endpoint-aware read path behave asexpected. ### For code changes: - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? *Note: Automated CI testing doesn't cover all cases so manual testing with cloud storage is still required.* - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? ### AI Tooling Contains content generated by GitHub Copilot. - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used. - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
