wsarecv commented on code in PR #3728:
URL: https://github.com/apache/gobblin/pull/3728#discussion_r1279914515
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -125,10 +126,13 @@ public Map<CompletenessType, Boolean>
calculateCompleteness(String datasetName,
countsByTier.forEach((x,y) -> log.info(String.format(" %s : %s ", x, y)));
Map<CompletenessType, Boolean> result = new HashMap<>();
- result.put(CompletenessType.ClassicCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.ClassicCompleteness, countsByTier) > threshold);
- result.put(CompletenessType.TotalCountCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.TotalCountCompleteness, countsByTier) > threshold);
+ Arrays.stream(CompletenessType.values()).forEach(type -> {
+ try {
+ result.put(type, calculateCompleteness(datasetName, beginInMillis,
endInMillis, type, countsByTier) > threshold);
+ } catch (IOException e) {
+ log.error("Failed to calculate completeness for type " + type, e);
Review Comment:
Should we set a value to false if its computation fails? So that the
returned map always contains two elements.
Just to avoid null when map.get() because `ClassicWatermarkUpdater` &
`TotalCountWatermarkUpdater` have similar logic like:
> protected void
computeAndUpdateInternal(Map<KafkaAuditCountVerifier.CompletenessType, Boolean>
results,
ZonedDateTime timestampDT) {
if
(!results.get(KafkaAuditCountVerifier.CompletenessType.TotalCountCompleteness))
{
return;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
