[
https://issues.apache.org/jira/browse/HADOOP-19474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938539#comment-17938539
]
ASF GitHub Bot commented on HADOOP-19474:
-----------------------------------------
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013904387
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1277,57 +1275,14 @@ public String listStatus(final Path path, final String
startFrom,
do {
try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) {
- AbfsRestOperation op = listingClient.listPath(relativePath, false,
- abfsConfiguration.getListMaxResults(), continuation,
- tracingContext);
+ ListResponseData listResponseData =
listingClient.listPath(relativePath,
+ false, abfsConfiguration.getListMaxResults(), continuation,
+ tracingContext, this.uri);
+ AbfsRestOperation op = listResponseData.getOp();
perfInfo.registerResult(op.getResult());
- continuation =
listingClient.getContinuationFromResponse(op.getResult());
- ListResultSchema retrievedSchema =
op.getResult().getListResultSchema();
- if (retrievedSchema == null) {
- throw new AbfsRestOperationException(
- AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
- AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
- "listStatusAsync path not found",
- null, op.getResult());
- }
-
- long blockSize = abfsConfiguration.getAzureBlockSize();
-
- for (ListResultEntrySchema entry : retrievedSchema.paths()) {
- final String owner =
identityTransformer.transformIdentityForGetRequest(entry.owner(), true,
userName);
- final String group =
identityTransformer.transformIdentityForGetRequest(entry.group(), false,
primaryUserGroup);
- final String encryptionContext = entry.getXMsEncryptionContext();
- final FsPermission fsPermission = entry.permissions() == null
- ? new AbfsPermission(FsAction.ALL, FsAction.ALL,
FsAction.ALL)
- : AbfsPermission.valueOf(entry.permissions());
- final boolean hasAcl =
AbfsPermission.isExtendedAcl(entry.permissions());
-
- long lastModifiedMillis = 0;
- long contentLength = entry.contentLength() == null ? 0 :
entry.contentLength();
- boolean isDirectory = entry.isDirectory() == null ? false :
entry.isDirectory();
- if (entry.lastModified() != null && !entry.lastModified().isEmpty())
{
- lastModifiedMillis = DateTimeUtils.parseLastModifiedTime(
- entry.lastModified());
- }
-
- Path entryPath = new Path(File.separator + entry.name());
- entryPath = entryPath.makeQualified(this.uri, entryPath);
-
- fileStatuses.add(
- new VersionedFileStatus(
- owner,
- group,
- fsPermission,
- hasAcl,
- contentLength,
- isDirectory,
- 1,
- blockSize,
- lastModifiedMillis,
- entryPath,
- entry.eTag(),
- encryptionContext));
- }
+ continuation = listResponseData.getContinuationToken();
+ List<FileStatus> fileStatusListInCurrItr =
listResponseData.getFileStatusList();
+ fileStatuses.addAll(fileStatusListInCurrItr);
Review Comment:
Good Suggestion taken
> ABFS: [FnsOverBlob] Listing Optimizations to avoid multiple iteration over
> list response.
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-19474
> URL: https://issues.apache.org/jira/browse/HADOOP-19474
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.5.0, 3.4.1
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
> Labels: pull-request-available
>
> On blob endpoint, there are a couple of handling that is needed to be done on
> client side.
> This involves:
> # Parsing of xml response and converting them to VersionedFileStatus list
> # Removing duplicate entries for non-empty explicit directories coming due
> to presence of the marker files
> # Trigerring Rename recovery on the previously failed rename indicated by
> the presence of pending json file.
> Currently all three are done in a separate iteration over whole list. This is
> to pbring all those things to a common place so that single iteration over
> list reposne can handle all three.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
