nsivabalan commented on code in PR #17940:
URL: https://github.com/apache/hudi/pull/17940#discussion_r2751771407
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java:
##########
@@ -387,6 +392,47 @@ public String validateFiles(
return HoodiePrintHelper.print(header, new HashMap<>(), "", false,
Integer.MAX_VALUE, false, rows);
}
+ // @ShellOption(value = "--backup", help = "Backup the metadata table before
delete", defaultValue = "true", arity = 1) final boolean backup
Review Comment:
why commented out. can we remove
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java:
##########
@@ -387,6 +392,47 @@ public String validateFiles(
return HoodiePrintHelper.print(header, new HashMap<>(), "", false,
Integer.MAX_VALUE, false, rows);
}
+ // @ShellOption(value = "--backup", help = "Backup the metadata table before
delete", defaultValue = "true", arity = 1) final boolean backup
+ @ShellMethod(key = "metadata lookup-record-index", value = "Print Record
index information for a record_key")
Review Comment:
lets fix the documentation to call out, either a record key for global RLI
or partition path and record key in case of partitioned RLI
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -1077,6 +1077,11 @@ public Builder withEnableGlobalRecordLevelIndex(boolean
enabled) {
return this;
}
+ public Builder withEnableRecordLevelIndexAtPartitionLevel(boolean enabled)
{
+ metadataConfig.setValue(RECORD_LEVEL_INDEX_ENABLE_PROP,
String.valueOf(enabled));
Review Comment:
We renamed the old ones.
we use
GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP
and RECORD_LEVEL_INDEX_ENABLE_PROP
for global variant and non global variant.
Lets do :
```
public Builder withEnableRecordLevelIndex(boolean enabled) {
metadataConfig.setValue(RECORD_LEVEL_INDEX_ENABLE_PROP,
String.valueOf(enabled));
return this;
}
```
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java:
##########
@@ -387,6 +392,47 @@ public String validateFiles(
return HoodiePrintHelper.print(header, new HashMap<>(), "", false,
Integer.MAX_VALUE, false, rows);
}
+ // @ShellOption(value = "--backup", help = "Backup the metadata table before
delete", defaultValue = "true", arity = 1) final boolean backup
+ @ShellMethod(key = "metadata lookup-record-index", value = "Print Record
index information for a record_key")
+ public String getRecordIndexInfo(
+ @ShellOption(value = "--record_key", help = "Record key entry whose info
will be fetched")
+ final String recordKey,
+ @ShellOption(value = "--partition_path_for_non_global_rli", help = "
Partition path needs to be provided for non Global or partition level Record
index",
+ defaultValue = "") final String partitionPath) {
+ HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+ HoodieStorage storage = metaClient.getStorage();
+ HoodieMetadataConfig config =
HoodieMetadataConfig.newBuilder().enable(true).build();
+ HoodieBackedTableMetadata metaReader = new HoodieBackedTableMetadata(
+ new HoodieLocalEngineContext(HoodieCLI.conf), storage, config,
HoodieCLI.basePath);
+
+ ValidationUtils.checkState(metaReader.enabled(), "[ERROR] Metadata Table
not enabled/initialized\n\n");
+
ValidationUtils.checkState(metaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX),
+ "[ERROR] Record index partition is not enabled/initialized\n\n");
+
+ Option<String> dataTablePartition =
StringUtils.isNullOrEmpty(partitionPath) ? Option.empty() :
Option.of(partitionPath);
+ HoodiePairData<String, HoodieRecordGlobalLocation>
recordKeyToGlobalLocationMap =
+
metaReader.readRecordIndexLocationsWithKeys(HoodieListData.eager(Collections.singletonList(recordKey)),
dataTablePartition);
+ List<Pair<String, HoodieRecordGlobalLocation>> recordLocationKeyPair =
recordKeyToGlobalLocationMap.collectAsList();
+ if (recordLocationKeyPair.isEmpty()) {
+ return "[INFO] Record key " + recordKey + " not found in Record Index";
Review Comment:
we should print the partition path as well incase of partitioned RLI
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java:
##########
@@ -387,6 +392,47 @@ public String validateFiles(
return HoodiePrintHelper.print(header, new HashMap<>(), "", false,
Integer.MAX_VALUE, false, rows);
}
+ // @ShellOption(value = "--backup", help = "Backup the metadata table before
delete", defaultValue = "true", arity = 1) final boolean backup
+ @ShellMethod(key = "metadata lookup-record-index", value = "Print Record
index information for a record_key")
+ public String getRecordIndexInfo(
+ @ShellOption(value = "--record_key", help = "Record key entry whose info
will be fetched")
+ final String recordKey,
+ @ShellOption(value = "--partition_path_for_non_global_rli", help = "
Partition path needs to be provided for non Global or partition level Record
index",
+ defaultValue = "") final String partitionPath) {
+ HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+ HoodieStorage storage = metaClient.getStorage();
+ HoodieMetadataConfig config =
HoodieMetadataConfig.newBuilder().enable(true).build();
+ HoodieBackedTableMetadata metaReader = new HoodieBackedTableMetadata(
+ new HoodieLocalEngineContext(HoodieCLI.conf), storage, config,
HoodieCLI.basePath);
+
+ ValidationUtils.checkState(metaReader.enabled(), "[ERROR] Metadata Table
not enabled/initialized\n\n");
+
ValidationUtils.checkState(metaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX),
+ "[ERROR] Record index partition is not enabled/initialized\n\n");
+
Review Comment:
we should check index definition and if its partitioned RLI, we need to
assert that `partition_path_for_non_global_rli` is not null or set by the user
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java:
##########
@@ -387,6 +392,47 @@ public String validateFiles(
return HoodiePrintHelper.print(header, new HashMap<>(), "", false,
Integer.MAX_VALUE, false, rows);
}
+ // @ShellOption(value = "--backup", help = "Backup the metadata table before
delete", defaultValue = "true", arity = 1) final boolean backup
+ @ShellMethod(key = "metadata lookup-record-index", value = "Print Record
index information for a record_key")
+ public String getRecordIndexInfo(
+ @ShellOption(value = "--record_key", help = "Record key entry whose info
will be fetched")
+ final String recordKey,
+ @ShellOption(value = "--partition_path_for_non_global_rli", help = "
Partition path needs to be provided for non Global or partition level Record
index",
Review Comment:
we can just name this `--partition_path`.
in the documentation/desc of the option, we can call out the purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
