dlmarion commented on code in PR #4750:
URL: https://github.com/apache/accumulo/pull/4750#discussion_r1690239324
##########
server/manager/src/main/java/org/apache/accumulo/manager/Manager.java:
##########
@@ -712,21 +708,35 @@ public void run() {
* migration will refer to a non-existing tablet, so it can never
complete. Periodically scan
* the metadata table and remove any migrating tablets that no longer
exist.
*/
- private void cleanupNonexistentMigrations(final AccumuloClient
accumuloClient)
+ private void cleanupNonexistentMigrations(final ClientContext
clientContext)
throws TableNotFoundException {
- Scanner scanner = accumuloClient.createScanner(MetadataTable.NAME,
Authorizations.EMPTY);
- TabletColumnFamily.PREV_ROW_COLUMN.fetch(scanner);
- scanner.setRange(MetadataSchema.TabletsSection.getRange());
- Set<KeyExtent> notSeen;
+
+ Map<DataLevel,Set<KeyExtent>> notSeen = new EnumMap<>(DataLevel.class);
+
synchronized (migrations) {
- notSeen = new HashSet<>(migrations.keySet());
+ // partition migrations by level
+ migrations.keySet().forEach(extent -> notSeen
+ .computeIfAbsent(DataLevel.of(extent.tableId()), k -> new
HashSet<>()).add(extent));
}
- for (Entry<Key,Value> entry : scanner) {
- KeyExtent extent = KeyExtent.fromMetaPrevRow(entry);
- notSeen.remove(extent);
+
+ // for each level find the set of migrating tablets that do not exists
in metadata store
+ for (DataLevel dataLevel : DataLevel.values()) {
+ var notSeenForLevel = notSeen.getOrDefault(dataLevel, Set.of());
+ if (notSeenForLevel.isEmpty() || dataLevel == DataLevel.ROOT) {
+ // No need to scan this level if there are no migrations. The root
tablet is always
+ // expected to exists, so no need to read its metadata.
+ continue;
+ }
+
+ try (var tablets =
clientContext.getAmple().readTablets().forLevel(dataLevel)
+ .fetch(TabletMetadata.ColumnType.PREV_ROW).build()) {
+ tablets.forEach(tabletMeta ->
notSeenForLevel.remove(tabletMeta.getExtent()));
Review Comment:
I talked with @keith-turner and I take back my suggestion. He said that this
would read all of the extents into memory, whereas using Ample would not as
it's using a Scanner underneath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
