aplex commented on a change in pull request #3394:
URL: https://github.com/apache/gobblin/pull/3394#discussion_r707791587
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java
##########
@@ -663,7 +663,7 @@ public void apply(JobListener jobListener, JobContext
jobContext)
@VisibleForTesting
public static long sumWorkUnitsSizes (WorkUnitStream workUnitStream) {
Collection<WorkUnit> workUnits =
JobLauncherUtils.flattenWorkUnits(workUnitStream.getMaterializedWorkUnitCollection());
- long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE)).sum();
+ long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE, 1)).sum();
Review comment:
I think we should fix this division by zero where it occurs. It's still
possible for the job to copy 0 bytes. For example, when it copies empty files.
In the place where we calculate progress, can we fallback to counting the
work units if total size is zero? So the variable says total size in bytes, and
it will confuse future developers if we put work unit count there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
