[
https://issues.apache.org/jira/browse/GOBBLIN-1862?focusedWorklogId=873613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-873613
]
ASF GitHub Bot logged work on GOBBLIN-1862:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jul/23 22:42
Start Date: 28/Jul/23 22:42
Worklog Time Spent: 10m
Work Description: vikrambohra commented on code in PR #3725:
URL: https://github.com/apache/gobblin/pull/3725#discussion_r1278137375
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -217,7 +217,7 @@ private static void validateTierCounts(String datasetName,
long beginInMillis, l
throw new IOException(String.format("Reference tier %s audit count
cannot be retrieved for dataset %s between %s and %s", refTier, datasetName,
beginInMillis, endInMillis));
}
long refCount = countsByTier.get(refTier);
- if(refCount <= 0) {
+ if (refCount < 0) {
throw new IOException(String.format("Reference tier %s count cannot be
less than or equal to zero", refTier));
Review Comment:
Change exception log line to match the if condition "less than zero"
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -217,7 +217,7 @@ private static void validateTierCounts(String datasetName,
long beginInMillis, l
throw new IOException(String.format("Reference tier %s audit count
cannot be retrieved for dataset %s between %s and %s", refTier, datasetName,
beginInMillis, endInMillis));
}
long refCount = countsByTier.get(refTier);
- if(refCount <= 0) {
+ if (refCount < 0) {
Review Comment:
I think we should add a log.warn when refCount == 0. wdyt?
##########
gobblin-completeness/src/test/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifierTest.java:
##########
@@ -113,6 +113,20 @@ public void testTotalCountCompleteness() throws
IOException {
));
Assert.assertFalse(verifier.calculateCompleteness(topic, 0L, 0L)
.get(KafkaAuditCountVerifier.CompletenessType.TotalCountCompleteness));
+
+ // Check validation tiers for exceptions
+ client.setTierCounts(ImmutableMap.of(
+ SOURCE_TIER, 990L,
+ REFERENCE_TIERS, 0L,
+ TOTAL_COUNT_REF_TIER_0, 0L,
+ TOTAL_COUNT_REF_TIER_1, 0L
+ ));
+ try {
+ verifier.calculateCompleteness(topic, 0L, 0L)
+
.get(KafkaAuditCountVerifier.CompletenessType.TotalCountCompleteness);
Review Comment:
Your test case only covers TotalCountCompleteness. Add a new test for
ClassicCompleteness
Issue Time Tracking
-------------------
Worklog Id: (was: 873613)
Remaining Estimate: 0h
Time Spent: 10m
> Allow reference count to be 0 but no less
> -----------------------------------------
>
> Key: GOBBLIN-1862
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1862
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Andy Jiang
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to possible issues upstream reporting incorrect reference counts for
> number of records, there could be times when the reference count reported to
> be 0.
> In the current case, if reference tier counter is reported to be 0, an
> exception is thrown, however is silently swallowed. Thus, causing the
> watermark to not move forward in this case or in the case of a quiet topic
> for hours where there are no records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
