[
https://issues.apache.org/jira/browse/GOBBLIN-2204?focusedWorklogId=977046&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-977046
]
ASF GitHub Bot logged work on GOBBLIN-2204:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 31/Jul/25 06:50
Start Date: 31/Jul/25 06:50
Worklog Time Spent: 10m
Work Description: vsinghal85 commented on code in PR #4113:
URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244505067
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java:
##########
@@ -613,6 +629,11 @@ private boolean checkDataQuality(Optional<Object> schema)
TaskLevelPolicyCheckResults taskResults =
this.taskContext.getTaskLevelPolicyChecker(this.forkTaskState,
this.branches > 1 ? this.index : -1)
.executePolicies();
+ boolean allRequiredPoliciesPassed =
taskResults.getPolicyResults().entrySet().stream()
+ .filter(e -> e.getValue() == TaskLevelPolicy.Type.FAIL)
+ .allMatch(e -> e.getKey() == TaskLevelPolicy.Result.PASSED);
+ forkTaskState.setProp(ConfigurationKeys.TASK_LEVEL_POLICY_RESULT_KEY,
+ allRequiredPoliciesPassed ? DataQualityStatus.PASSED.name() :
DataQualityStatus.FAILED.name());
Review Comment:
This is eventually what above statement does, it filters out all the
required policies in the first step, and then checks whether all these required
policies had their data quality result passed or not, if any policy would fail,
it would result in overall failure of fork data quality.
Issue Time Tracking
-------------------
Worklog Id: (was: 977046)
Time Spent: 1h 50m (was: 1h 40m)
> FileSize Data Quality implementation for FileBasedCopy
> ------------------------------------------------------
>
> Key: GOBBLIN-2204
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2204
> Project: Apache Gobblin
> Issue Type: Task
> Reporter: Vaibhav Singhal
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
