Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
ashishkumar50 commented on PR #10145: URL: https://github.com/apache/ozone/pull/10145#issuecomment-4378408600 Thanks @sreejasahithi for the patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
ashishkumar50 merged PR #10145: URL: https://github.com/apache/ozone/pull/10145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
sreejasahithi commented on code in PR #10145:
URL: https://github.com/apache/ozone/pull/10145#discussion_r3159853381
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
Review Comment:
When startMetadata is not provided, deltaSnapshots will be equal to the full
set of snapshots collected across all version files during the version file
rewrite phase. When startMetadata is provided, deltaSnapshots will contain only
those snapshots that are not tracked by the start version i.e. snapshots that
appeared in intermediate version files between start and end, minus the
snapshots already present in the start version's metadata.
Because deltaSnapshots is built by reading each intermediate version file's
JSON as it was re-written at that point in time, it can include snapshots that
were subsequently expired.
So we don't use deltaSnapshots for iterating and instead iterate through the
snapshots collected from the endVersion metadata file because we won't be able
to read the manifest list associated with the expired snapshots.
In `manifestsToRewrite` we use deltaSnapshots only to avoid including
manifest files that were already rewritten in a previous incremental run. The
snapshot_id field on each manifest entry identifies the snapshot that
originally created it. By filtering to only those whose snapshot_id falls
within deltaSnapshotIds, we select only manifests that are new since the start
version and exclude those that were inherited from before it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
sreejasahithi commented on PR #10145: URL: https://github.com/apache/ozone/pull/10145#issuecomment-4368603984 @ashishkumar50 I have updated the patch, could you please review it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
ashishkumar50 commented on code in PR #10145:
URL: https://github.com/apache/ozone/pull/10145#discussion_r3179397089
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
Review Comment:
In that case `manifestsToRewrite` you can populate outside the caller and
avoid passing startMetadata here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
sreejasahithi commented on PR #10145: URL: https://github.com/apache/ozone/pull/10145#issuecomment-4342743231 > @sreejasahithi Thanks for the PR, please find the comments inline. Also can you add test for this change. We can add the tests once we implement manifest-list rewrite because manifest-list rewrite will use manifestsToRewrite result to update the copy plan , so it will be easier to test at that time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
sreejasahithi commented on code in PR #10145:
URL: https://github.com/apache/ozone/pull/10145#discussion_r3159905459
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
+ deltaSnapshotIds =
deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());
+} else {
+ deltaSnapshotIds = null;
+}
+
+Set manifestPaths = ConcurrentHashMap.newKeySet();
+int maxInFlight = parallelism * MAX_INFLIGHT_MULTIPLIER;
+Semaphore semaphore = new Semaphore(maxInFlight);
+
+ExecutorCompletionService completionService = new
ExecutorCompletionService<>(executorService);
+
+int submittedTasks = 0;
+int completedTasks = 0;
+
+try {
+ for (Snapshot snapshot : endStaticTable.snapshots()) {
+semaphore.acquire(); // blocks when maxInFlight tasks are already
in-flight
+
+final long snapshotId = snapshot.snapshotId();
+final String manifestListLocation = snapshot.manifestListLocation();
+
+boolean taskSubmitted = false;
+try {
+ completionService.submit(() -> {
+try (CloseableIterable manifests =
+ InternalData.read(
+ FileFormat.AVRO,
+ table.io().newInputFile(manifestListLocation))
+ .setRootType(GenericManifestFile.class)
+ .setCustomType(
+ ManifestFile.PARTITION_SUMMARIES_ELEMENT_ID,
+ GenericPartitionFieldSummary.class)
+ .project(ManifestFile.schema())
+ .build()) {
+
+ for (ManifestFile manifest : manifests) {
+if (deltaSnapshotIds == null) {
+ manifestPaths.add(manifest.path());
+} else if (manifest.snapshotId() != null
Review Comment:
For any manifest list file that was correctly written, snapshotId() will
never be null.
The check added here is a defensive check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
sreejasahithi commented on code in PR #10145:
URL: https://github.com/apache/ozone/pull/10145#discussion_r3159853381
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
Review Comment:
When startMetadata is not provided, deltaSnapshots will be equal to the full
set of snapshots collected across all version files during the version file
rewrite phase. When startMetadata is provided, deltaSnapshots will contain only
those snapshots that are not tracked by the start version i.e. snapshots that
appeared in intermediate version files between start and end, minus the
snapshots already present in the start version's metadata.
Because deltaSnapshots is built by reading each intermediate version file's
JSON as it was written at that point in time, it can include snapshots that
were subsequently expired.
So we don't use deltaSnapshots for iterating and instead iterate through the
snapshots collected from the endVersion metadata file because we won't be able
to read the manifest list associated with the expired snapshots.
In `manifestsToRewrite` we use deltaSnapshots only to avoid including
manifest files that were already rewritten in a previous incremental run. The
snapshot_id field on each manifest entry identifies the snapshot that
originally created it. By filtering to only those whose snapshot_id falls
within deltaSnapshotIds, we select only manifests that are new since the start
version and exclude those that were inherited from before it.
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -57,11 +69,15 @@ public class RewriteTablePathOzoneAction implements
RewriteTablePath {
private String stagingDir;
private int parallelism;
+ private ExecutorService executorService;
+ private static final int MAX_INFLIGHT_MULTIPLIER = 4;
+ private static final int DEFAULT_THREAD_COUNT = 10;
+
private final Table table;
public RewriteTablePathOzoneAction(Table table) {
this.table = table;
-this.parallelism = Runtime.getRuntime().availableProcessors();
+this.parallelism = DEFAULT_THREAD_COUNT;
Review Comment:
Yes , we can pass the thread count via command during which
` public RewriteTablePathOzoneAction(Table table, int parallelism)` will be
used , we can add this when the CLI command is added for the rewrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-14942. Implement manifest selection logic for rewrite based on snapshot delta [ozone]
ashishkumar50 commented on code in PR #10145:
URL: https://github.com/apache/ozone/pull/10145#discussion_r3158697855
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
Review Comment:
The entire `startMetadata` object is passed but only .equals(null) is
checked on it. The actual snapshot data is already captured in deltaSnapshots.
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -251,4 +271,109 @@ private Set>
rewriteVersionFile(TableMetadata metadata, Str
return result;
}
+
+ private Set manifestsToRewrite(Set deltaSnapshots,
TableMetadata startMetadata) {
+Table endStaticTable =
RewriteTablePathOzoneUtils.newStaticTable(endVersionName, table.io());
+
+final Set deltaSnapshotIds;
+if (startMetadata != null) {
+ deltaSnapshotIds =
deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());
+} else {
+ deltaSnapshotIds = null;
+}
+
+Set manifestPaths = ConcurrentHashMap.newKeySet();
+int maxInFlight = parallelism * MAX_INFLIGHT_MULTIPLIER;
+Semaphore semaphore = new Semaphore(maxInFlight);
+
+ExecutorCompletionService completionService = new
ExecutorCompletionService<>(executorService);
+
+int submittedTasks = 0;
+int completedTasks = 0;
+
+try {
+ for (Snapshot snapshot : endStaticTable.snapshots()) {
+semaphore.acquire(); // blocks when maxInFlight tasks are already
in-flight
+
+final long snapshotId = snapshot.snapshotId();
+final String manifestListLocation = snapshot.manifestListLocation();
+
+boolean taskSubmitted = false;
+try {
+ completionService.submit(() -> {
+try (CloseableIterable manifests =
+ InternalData.read(
+ FileFormat.AVRO,
+ table.io().newInputFile(manifestListLocation))
+ .setRootType(GenericManifestFile.class)
+ .setCustomType(
+ ManifestFile.PARTITION_SUMMARIES_ELEMENT_ID,
+ GenericPartitionFieldSummary.class)
+ .project(ManifestFile.schema())
+ .build()) {
+
+ for (ManifestFile manifest : manifests) {
+if (deltaSnapshotIds == null) {
+ manifestPaths.add(manifest.path());
+} else if (manifest.snapshotId() != null
Review Comment:
Legacy or old Iceberg manifests can have snapshotId() == null ?
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -205,6 +223,8 @@ private String rebuildMetadata() {
}
RewriteResult rewriteVersionResult =
rewriteVersionFiles(endMetadata);
+Set deltaSnapshots = deltaSnapshots(startMetadata,
rewriteVersionResult.toRewrite());
+Set manifestsToRewrite = manifestsToRewrite(deltaSnapshots,
startMetadata);
Review Comment:
Add a TODO here what `manifestsToRewrite` will be used for
##
hadoop-ozone/iceberg/src/main/java/org/apache/hadoop/ozone/iceberg/RewriteTablePathOzoneAction.java:
##
@@ -57,11 +69,15 @@ public class RewriteTablePathOzoneAction implements
RewriteTablePath {
private String stagingDir;
private int parallelism;
+ private ExecutorService executorService;
+ private static final int MAX_INFLIGHT_MULTIPLIER = 4;
+ private static final int DEFAULT_THREAD_COUNT = 10;
+
private final Table table;
public RewriteTablePathOzoneAction(Table table) {
this.table = table;
-this.parallelism = Runtime.getRuntime().availableProcessors();
+this.parallelism = DEFAULT_THREAD_COUNT;
Review Comment:
Thread count can be passed via command, can be done in subsequent PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
