[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r512452877
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2276,318 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+// unsorted OMKeyInfo list contains combine results from TableCache and DB.
+List fileStatusFinalList = new ArrayList<>();
+LinkedHashSet fileStatusList = new LinkedHashSet<>();
+if (numEntries <= 0) {
+ return fileStatusFinalList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+String seekFileInDB;
+String seekDirInDB;
+long prefixKeyInDB;
+String prefixPath = keyName;
+
+int countEntries = 0;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// Not required to search in DeletedTable because all the deleted
+// keys will be marked directly in dirTable or in keyTable by
+// breaking the pointer to its sub-dirs. So, there is no issue of
+// inconsistency.
+
+/*
+ * keyName is a directory.
+ * Say, "/a" is the dir name and its objectID is 1024, then seek
+ * will be doing with "1024/" to get all immediate descendants.
+ */
+if (fileStatus.getKeyInfo() != null) {
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ // list root directory.
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+seekFileInDB = metadataManager.getOzonePathKey(prefixKeyInDB, "");
+seekDirInDB = metadataManager.getOzonePathKey(prefixKeyInDB, "");
+
+// TODO: recursive flag=true will be handled in HDDS-4360 jira.
+// Order of seek -> (1)Seek dirs in dirTable (2)Seek files in fileTable
+// 1. Seek the given key in key table.
+countEntries = getFilesFromDirectory(fileStatusList, seekFileInDB,
+prefixPath, prefixKeyInDB, startKey, countEntries, numEntries);
+// 2. Seek the given key in dir table.
+getDirectories(recursive, startKey, numEntries, fileStatusList,
+volumeName, bucketName, seekDirInDB, prefixKeyInDB,
+prefixPath, countEntries);
+ } else {
+/*
+ * startKey will be used in iterator seek and sets the beginning point
+ * for key traversal.
+ *
+ * key name will be used as parentID where the user has requested to
+ * list the keys from.
+ *
+ * When recursive flag=false, parentID won't change between two pages.
+ * For example: OM has a namespace like,
+ */a/1...1M files and /a/b/1...1M files.
+ */a/1...1M directories and /a/b/1...1M directories.
+ * Listing "/a", will always have the parentID as "a" irrespective of
+ * the startKey value.
+ */
+// TODO: recursive flag=true will be handled in HDDS-4360 jira.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null, true);
+
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ if(fileStatusInfo.isDirectory()){
+seekDirInDB = metadataManager.getOzonePathKey(prefixKeyInDB,
+fileStatusInfo.getKeyInfo().getFileName());
+
+// Order of seek -> (1) Seek dirs in dirTable. In OM, always the
+// order of search is, first seek into fileTable and then dirTable.
+// So, its not required to search again into the fileTable.
+
+// Seek the given key in dirTable.
+getDirectories(recursive, startKey, numEntries,
+fileStatusList, volumeName, bucketName, seekDirInDB,
+prefixKeyInDB, prefixPath, countEntries);
+
+ } else {
+seekFileInDB = metadataManager.getOzonePathKey(prefixKeyInDB,
Review comment:
As we have to perform seek in two tables to finish the listing, I'm
maintaining a seek order
1) Seek all the files from
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r512452877
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2276,318 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+// unsorted OMKeyInfo list contains combine results from TableCache and DB.
+List fileStatusFinalList = new ArrayList<>();
+LinkedHashSet fileStatusList = new LinkedHashSet<>();
+if (numEntries <= 0) {
+ return fileStatusFinalList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+String seekFileInDB;
+String seekDirInDB;
+long prefixKeyInDB;
+String prefixPath = keyName;
+
+int countEntries = 0;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// Not required to search in DeletedTable because all the deleted
+// keys will be marked directly in dirTable or in keyTable by
+// breaking the pointer to its sub-dirs. So, there is no issue of
+// inconsistency.
+
+/*
+ * keyName is a directory.
+ * Say, "/a" is the dir name and its objectID is 1024, then seek
+ * will be doing with "1024/" to get all immediate descendants.
+ */
+if (fileStatus.getKeyInfo() != null) {
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ // list root directory.
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+seekFileInDB = metadataManager.getOzonePathKey(prefixKeyInDB, "");
+seekDirInDB = metadataManager.getOzonePathKey(prefixKeyInDB, "");
+
+// TODO: recursive flag=true will be handled in HDDS-4360 jira.
+// Order of seek -> (1)Seek dirs in dirTable (2)Seek files in fileTable
+// 1. Seek the given key in key table.
+countEntries = getFilesFromDirectory(fileStatusList, seekFileInDB,
+prefixPath, prefixKeyInDB, startKey, countEntries, numEntries);
+// 2. Seek the given key in dir table.
+getDirectories(recursive, startKey, numEntries, fileStatusList,
+volumeName, bucketName, seekDirInDB, prefixKeyInDB,
+prefixPath, countEntries);
+ } else {
+/*
+ * startKey will be used in iterator seek and sets the beginning point
+ * for key traversal.
+ *
+ * key name will be used as parentID where the user has requested to
+ * list the keys from.
+ *
+ * When recursive flag=false, parentID won't change between two pages.
+ * For example: OM has a namespace like,
+ */a/1...1M files and /a/b/1...1M files.
+ */a/1...1M directories and /a/b/1...1M directories.
+ * Listing "/a", will always have the parentID as "a" irrespective of
+ * the startKey value.
+ */
+// TODO: recursive flag=true will be handled in HDDS-4360 jira.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null, true);
+
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ if(fileStatusInfo.isDirectory()){
+seekDirInDB = metadataManager.getOzonePathKey(prefixKeyInDB,
+fileStatusInfo.getKeyInfo().getFileName());
+
+// Order of seek -> (1) Seek dirs in dirTable. In OM, always the
+// order of search is, first seek into fileTable and then dirTable.
+// So, its not required to search again into the fileTable.
+
+// Seek the given key in dirTable.
+getDirectories(recursive, startKey, numEntries,
+fileStatusList, volumeName, bucketName, seekDirInDB,
+prefixKeyInDB, prefixPath, countEntries);
+
+ } else {
+seekFileInDB = metadataManager.getOzonePathKey(prefixKeyInDB,
Review comment:
As we have to perform seek in two tables to finish the listing, I'm
maintaining a seek order
1) Finish listing all the files
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509846900
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
Review comment:
Raised HDDS-4364 jira to handle this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509548807
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ seekKeyInDB = prefixKeyInDB + OZONE_URI_DELIMITER
+ + fileStatusInfo.getKeyInfo().getFileName();
+}
+ }
+
+ // Not required to search in TableCache because all the deleted keys
+ // are marked directly in directory table or in key table by breaking
+ // the pointer to its sub-dirs. So, there is no issue of inconsistency.
+ int countEntries = 0;
Review comment:
Thanks @linyiqun for pointing out this case. I have implemented File &
Dir Table cache logic in latest commit. As the plan is to update the Dir &
FileTable directly during delete api, ideally there is no need of considering
deletedTableCache. Will revisit this logic once we implement the delete api.
I have commented in [HDDS-4358
jira](https://issues.apache.org/jira/browse/HDDS-4358?focusedCommentId=17218477=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17218477)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509543716
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ seekKeyInDB = prefixKeyInDB + OZONE_URI_DELIMITER
+ + fileStatusInfo.getKeyInfo().getFileName();
+}
+ }
+
+ // Not required to search in TableCache because all the deleted keys
+ // are marked directly in directory table or in key table by breaking
+ // the pointer to its sub-dirs. So, there is no issue of inconsistency.
+ int countEntries = 0;
+ // Seek the given key in key table.
+ countEntries = getFilesFromDirectory(cacheKeyMap, seekKeyInDB,
+ prefixPath, countEntries, numEntries, prefixKeyInDB);
+ // Seek the given key in dir table.
+ Table dirTable = metadataManager.getDirectoryTable();
+ TableIterator>
+ iterator = dirTable.iterator();
+
+ iterator.seek(seekKeyInDB);
+
+ while (iterator.hasNext() && numEntries - countEntries > 0) {
+String entryInDb = iterator.key();
+OmDirectoryInfo dirInfo = iterator.value().getValue();
+if (!isImmediateChild(dirInfo.getParentObjectID(), prefixKeyInDB)) {
+ break;
+}
+
+if (recursive) {
+ // for recursive list all the entries
+ prefixPath = OMFileRequest.getAbsolutePath(prefixPath,
+ dirInfo.getName());
+ OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfo(volumeName,
+ bucketName, dirInfo, prefixPath);
+ cacheKeyMap.put(entryInDb,
+ new OzoneFileStatus(omKeyInfo, 0, true));
+ ++countEntries;
+ // files from this directory
+ seekKeyInDB = dirInfo.getObjectID() + OZONE_URI_DELIMITER;
Review comment:
Done
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -583,4 +588,112 @@ public static OmKeyInfo getOmKeyInfoFromFileTable(boolean
openFileTable,
return dbOmKeyInfo;
}
+ /**
+ * Gets OmKeyInfo if exists for the given key name in the DB.
+ *
+ * @param omMetadataMgr metadata manager
+ * @param volumeNamevolume name
+ * @param bucketNamebucket name
+ * @param keyName key name
+ * @param scmBlockSize scm block size
+ * @return OzoneFileStatus
+ * @throws IOException DB failure
+ */
+ @Nullable
+ public static OzoneFileStatus getOMKeyInfoIfExists(
+ OMMetadataManager omMetadataMgr, String volumeName, String bucketName,
+ String keyName, long scmBlockSize) throws IOException
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509543376
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
Review comment:
Good point. I kept this as an open point now and not addressed in my
latest commit. Need to do some more testing to understand the existing behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509543538
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ seekKeyInDB = prefixKeyInDB + OZONE_URI_DELIMITER
+ + fileStatusInfo.getKeyInfo().getFileName();
+}
+ }
+
+ // Not required to search in TableCache because all the deleted keys
+ // are marked directly in directory table or in key table by breaking
+ // the pointer to its sub-dirs. So, there is no issue of inconsistency.
+ int countEntries = 0;
+ // Seek the given key in key table.
+ countEntries = getFilesFromDirectory(cacheKeyMap, seekKeyInDB,
+ prefixPath, countEntries, numEntries, prefixKeyInDB);
+ // Seek the given key in dir table.
+ Table dirTable = metadataManager.getDirectoryTable();
+ TableIterator>
+ iterator = dirTable.iterator();
+
+ iterator.seek(seekKeyInDB);
+
+ while (iterator.hasNext() && numEntries - countEntries > 0) {
+String entryInDb = iterator.key();
+OmDirectoryInfo dirInfo = iterator.value().getValue();
+if (!isImmediateChild(dirInfo.getParentObjectID(), prefixKeyInDB)) {
+ break;
+}
+
+if (recursive) {
+ // for recursive list all the entries
+ prefixPath = OMFileRequest.getAbsolutePath(prefixPath,
+ dirInfo.getName());
+ OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfo(volumeName,
+ bucketName, dirInfo, prefixPath);
+ cacheKeyMap.put(entryInDb,
+ new OzoneFileStatus(omKeyInfo, 0, true));
+ ++countEntries;
+ // files from this directory
+ seekKeyInDB = dirInfo.getObjectID() + OZONE_URI_DELIMITER;
+ countEntries = getFilesFromDirectory(cacheKeyMap, seekKeyInDB,
+ prefixPath, countEntries, numEntries, prefixKeyInDB);
+} else {
+ String dirName = OMFileRequest.getAbsolutePath(prefixPath,
+ dirInfo.getName());
+ OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfo(volumeName,
+ bucketName, dirInfo, dirName);
+ cacheKeyMap.put(entryInDb,
+ new OzoneFileStatus(omKeyInfo, 0, true));
+ countEntries++;
+}
+// move to next entry in the table
+iterator.next();
+ }
+} finally {
+ metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
+ bucketName);
+}
+
+int countEntries = 0;
+// Convert results in cacheKeyMap to List
+for (Map.Entry entry :
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509535852
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ seekKeyInDB = prefixKeyInDB + OZONE_URI_DELIMITER
+ + fileStatusInfo.getKeyInfo().getFileName();
+}
+ }
+
+ // Not required to search in TableCache because all the deleted keys
+ // are marked directly in directory table or in key table by breaking
+ // the pointer to its sub-dirs. So, there is no issue of inconsistency.
+ int countEntries = 0;
+ // Seek the given key in key table.
+ countEntries = getFilesFromDirectory(cacheKeyMap, seekKeyInDB,
+ prefixPath, countEntries, numEntries, prefixKeyInDB);
+ // Seek the given key in dir table.
+ Table dirTable = metadataManager.getDirectoryTable();
+ TableIterator>
+ iterator = dirTable.iterator();
+
+ iterator.seek(seekKeyInDB);
+
+ while (iterator.hasNext() && numEntries - countEntries > 0) {
+String entryInDb = iterator.key();
+OmDirectoryInfo dirInfo = iterator.value().getValue();
+if (!isImmediateChild(dirInfo.getParentObjectID(), prefixKeyInDB)) {
+ break;
+}
+
+if (recursive) {
+ // for recursive list all the entries
+ prefixPath = OMFileRequest.getAbsolutePath(prefixPath,
+ dirInfo.getName());
+ OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfo(volumeName,
+ bucketName, dirInfo, prefixPath);
+ cacheKeyMap.put(entryInDb,
+ new OzoneFileStatus(omKeyInfo, 0, true));
+ ++countEntries;
+ // files from this directory
+ seekKeyInDB = dirInfo.getObjectID() + OZONE_URI_DELIMITER;
+ countEntries = getFilesFromDirectory(cacheKeyMap, seekKeyInDB,
+ prefixPath, countEntries, numEntries, prefixKeyInDB);
+} else {
+ String dirName = OMFileRequest.getAbsolutePath(prefixPath,
+ dirInfo.getName());
+ OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfo(volumeName,
+ bucketName, dirInfo, dirName);
+ cacheKeyMap.put(entryInDb,
+ new OzoneFileStatus(omKeyInfo, 0, true));
+ countEntries++;
+}
+// move to next entry in the table
+iterator.next();
+ }
+} finally {
+ metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
+ bucketName);
+}
+
+int countEntries = 0;
+// Convert results in cacheKeyMap to List
+for (Map.Entry entry :
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r509534970
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1831,6 +1838,62 @@ private OzoneFileStatus getOzoneFileStatus(String
volumeName,
FILE_NOT_FOUND);
}
+
+ private OzoneFileStatus getOzoneFileStatusV1(String volumeName,
+ String bucketName,
+ String keyName,
+ boolean sortDatanodes,
+ String clientAddress)
+ throws IOException {
+OzoneFileStatus fileStatus = null;
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ // Check if this is the root of the filesystem.
+ if (keyName.length() == 0) {
+validateBucket(volumeName, bucketName);
+return new OzoneFileStatus();
+ }
+
+ fileStatus = OMFileRequest.getOMKeyInfoIfExists(metadataManager,
+ volumeName, bucketName, keyName, scmBlockSize);
+
+ // Check if the key is a directory.
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508999204
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
Review comment:
Yes, this condition is for the root directory listing.
`fileSystem.listStatus(new Path("o3fs://bucket1.vol1/"))`
For a wrong or non-existing keyname, today it is throwing exception. I have
referred code base
[KeyManagerImpl.java#L1823](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1823)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508953520
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
Review comment:
Noted and will add TODO in V1 code. I will watch #1451 progress and will
incorporate that logic in V1 once that PR is merged. Hope that make sense to
you?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508673206
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
Review comment:
Thanks a lot for pointing out this. Since the startKey value is
internally taken from the OMKeyInfo.keyName, I thought the path would have
already been in the normalized form. I saw startKey normalization in PR #1451,
I would like to hear from @bharatviswa504 about the plans and follow the same
logic here as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508666092
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
+}
+ } else {
+// startKey will be used in iterator seek and sets the beginning point
+// for key traversal.
+// key name will be used as parent ID where the user has requested to
+// list the keys from.
+OzoneFileStatus fileStatusInfo = getOzoneFileStatusV1(volumeName,
+bucketName, startKey, false, null);
+if (fileStatusInfo != null) {
+ prefixKeyInDB = fileStatusInfo.getKeyInfo().getParentObjectID();
+ seekKeyInDB = prefixKeyInDB + OZONE_URI_DELIMITER
+ + fileStatusInfo.getKeyInfo().getFileName();
Review comment:
Good catch. With recursive flag false, prefix path will always be same
as user given keyname. But with recursive flag true, it can change between two
pages, if the second page is iterating on the sub-dirs of the given user
keyname.
I have raised separate jira HDDS-4360 to handle the recursive logic. IMHO
that would be helpful to do more focussed development.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508418142
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2205,6 +2272,167 @@ private void listStatusFindKeyInTableCache(
return fileStatusList;
}
+ public List listStatusV1(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries, String clientAddress)
+ throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+
+List fileStatusList = new ArrayList<>();
+if (numEntries <= 0) {
+ return fileStatusList;
+}
+
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+// A map sorted by OmKey to combine results from TableCache and DB.
+TreeMap cacheKeyMap = new TreeMap<>();
+String seekKeyInDB = "";
+long prefixKeyInDB = Long.MIN_VALUE;
+String prefixPath = keyName;
+
+metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
+bucketName);
+try {
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args, clientAddress);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+
+// keyName is a directory
+if (fileStatus.getKeyInfo() != null) {
+ seekKeyInDB = fileStatus.getKeyInfo().getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = fileStatus.getKeyInfo().getObjectID();
+} else {
+ String bucketKey = metadataManager.getBucketKey(volumeName,
+ bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().get(bucketKey);
+ seekKeyInDB = omBucketInfo.getObjectID()
+ + OZONE_URI_DELIMITER;
+ prefixKeyInDB = omBucketInfo.getObjectID();
Review comment:
"/vol/bucket/" is the root directory and the first level directories are
under this. Thats the reason I left, `prefixPath` as empty. PrefixPath is used
to build the absolute key path name which will be added to the
OMKeyInfo#keyName. Existing code base don't have `volume/bucket name` in the
key path name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
