(accumulo) branch elasticity updated: avoids a metadata lookup per mutation for batch writer (#4280)

[kturner]

This is an automated email from the ASF dual-hosted git repository.

kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/elasticity by this push:
     new af9b0602db avoids a metadata lookup per mutation for batch writer 
(#4280)
af9b0602db is described below

commit af9b0602dbb3debb34828d7d3116ea310e175163
Author: Keith Turner <ktur...@apache.org>
AuthorDate: Fri Feb 16 19:01:03 2024 -0500

    avoids a metadata lookup per mutation for batch writer (#4280)
    
    When writing to unhosted ondemand tablets the batch writer was performing
    a metadata lookup per mutation before it requested hosting.  This was of
    course extremely slow.  This commit makes a quick change to perform a
    metadata lookup per extent instead of per mutation.  This could be
    further improved because its still not as good as the pre elasticity
    code. There is in ELASTICITY_TODO in the commit about further
    improvements.
    
    This is a massive improvement.  Was running FateStarvationIT which
    writes 100K rows.  Without this change it took 53 secs.  With this
    change it took 1.7 secs.
    
    This mostly solves #3708, there may be some further minor improvements
    that could be made.
---
 .../accumulo/core/clientImpl/ClientTabletCacheImpl.java   | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
index 4539a60b3a..10fb3aa21e 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
@@ -234,12 +234,23 @@ public class ClientTabletCacheImpl extends 
ClientTabletCache {
 
       wLock.lock();
       try {
+        CachedTablet lastTablet = null;
         for (T mutation : notInCache) {
 
           row.set(mutation.getRow());
 
-          CachedTablet tl =
-              _findTablet(context, row, false, false, false, lcSession, 
LocationNeed.REQUIRED);
+          // ELASTICITY_TODO using lastTablet avoids doing a metadata table 
lookup per mutation.
+          // However this still does at least one metadata lookup per tablet. 
This is not as good as
+          // the pre-elasticity code that would lookup N tablets at once and 
use them to bin
+          // mutations. So there is further room for improvement in the way 
this code interacts with
+          // cache and metadata table.
+          CachedTablet tl;
+          if (lastTablet != null && lastTablet.getExtent().contains(row)) {
+            tl = lastTablet;
+          } else {
+            tl = _findTablet(context, row, false, false, false, lcSession, 
LocationNeed.REQUIRED);
+            lastTablet = tl;
+          }
 
           if (!addMutation(binnedMutations, mutation, tl, lcSession)) {
             failures.add(mutation);