[
https://issues.apache.org/jira/browse/GOBBLIN-1864?focusedWorklogId=873907&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-873907
]
ASF GitHub Bot logged work on GOBBLIN-1864:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 31/Jul/23 21:54
Start Date: 31/Jul/23 21:54
Worklog Time Spent: 10m
Work Description: wsarecv commented on code in PR #3728:
URL: https://github.com/apache/gobblin/pull/3728#discussion_r1279914515
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -125,10 +126,13 @@ public Map<CompletenessType, Boolean>
calculateCompleteness(String datasetName,
countsByTier.forEach((x,y) -> log.info(String.format(" %s : %s ", x, y)));
Map<CompletenessType, Boolean> result = new HashMap<>();
- result.put(CompletenessType.ClassicCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.ClassicCompleteness, countsByTier) > threshold);
- result.put(CompletenessType.TotalCountCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.TotalCountCompleteness, countsByTier) > threshold);
+ Arrays.stream(CompletenessType.values()).forEach(type -> {
+ try {
+ result.put(type, calculateCompleteness(datasetName, beginInMillis,
endInMillis, type, countsByTier) > threshold);
+ } catch (IOException e) {
+ log.error("Failed to calculate completeness for type " + type, e);
Review Comment:
Should we set a value to false if its computation fails? So that the
returned map always contains two elements.
Just to avoid null when map.get(), `ClassicWatermarkUpdater` &
`TotalCountWatermarkUpdater` have similar logic like:
> protected void
computeAndUpdateInternal(Map<KafkaAuditCountVerifier.CompletenessType, Boolean>
results,
ZonedDateTime timestampDT) {
if
(!results.get(KafkaAuditCountVerifier.CompletenessType.TotalCountCompleteness))
{
return;
}
Issue Time Tracking
-------------------
Worklog Id: (was: 873907)
Time Spent: 40m (was: 0.5h)
> Catch exceptions if specific watermark calculation fails
> --------------------------------------------------------
>
> Key: GOBBLIN-1864
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1864
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
