voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863085631
##########
hudi-io/src/main/java/org/apache/hudi/common/metrics/Registry.java:
##########
@@ -190,6 +199,28 @@ static void setRegistries(Collection<Registry> registries)
{
*/
void set(String name, long value);
+ /**
+ * Subtract a set of counts previously read out of this registry, clamping
every counter at zero.
+ *
+ * Used to hand a batch of counters over to a consumer that owns them from
then on -- the commit-boundary
+ * drain for the record index lookup counters -- without discarding whatever
arrived after they were read.
+ *
+ * Clamping is what distinguishes this from {@code add(name, -value)}. The
registry can be emptied
+ * underneath a caller by an unrelated destructive scrape ({@link
#getAllMetrics(boolean, boolean)} with
+ * {@code flush=true} clears every registry in the process), and an
unbounded subtraction would then leave
+ * negative counters behind for good.
+ *
+ * The default is a best-effort read-modify-write. Implementations able to
do this atomically should
+ * override it, and should drop counters that reach zero rather than leaving
them at zero, so a registry
+ * nobody is writing to reads as empty.
+ *
+ * @param counts the counts to release, as returned by {@link
#getAllCounts(boolean)}.
+ */
+ default void release(Map<String, Long> counts) {
Review Comment:
Tracked on #19759 (section B) with the rest of the hudi-io lookup surface;
not blocking here.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDReadClient.java:
##########
@@ -210,8 +212,16 @@ public JavaRDD<HoodieRecord<T>>
filterExists(JavaRDD<HoodieRecord<T>> hoodieReco
* @return Tagged RDD of Hoodie records
*/
public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>>
hoodieRecords) throws HoodieIndexException {
- return HoodieJavaRDD.getJavaRDD(
- index.tagLocation(HoodieJavaRDD.of(hoodieRecords), context,
hoodieTable));
+ // Lookups driven from the read client are dedupe traffic, not
tag-location traffic. Label them so
+ // the two are distinguishable in the reported counters. Driver-side only:
the label is
+ // captured when the lookup closure is built.
+ String previousCaller =
RecordIndexLookupMetrics.setCaller(RecordIndexMetricNames.CALLER_DEDUPE);
Review Comment:
Tracked on #19759 (section C) for restoration with coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
