[
https://issues.apache.org/jira/browse/GOBBLIN-1543?focusedWorklogId=650238&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650238
]
ASF GitHub Bot logged work on GOBBLIN-1543:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Sep/21 21:28
Start Date: 13/Sep/21 21:28
Worklog Time Spent: 10m
Work Description: umustafi commented on a change in pull request #3394:
URL: https://github.com/apache/gobblin/pull/3394#discussion_r707693945
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java
##########
@@ -663,7 +663,7 @@ public void apply(JobListener jobListener, JobContext
jobContext)
@VisibleForTesting
public static long sumWorkUnitsSizes (WorkUnitStream workUnitStream) {
Collection<WorkUnit> workUnits =
JobLauncherUtils.flattenWorkUnits(workUnitStream.getMaterializedWorkUnitCollection());
- long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE)).sum();
+ long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE, 1)).sum();
Review comment:
We had a case where a user started a retention job with progress
reporting that failed since the Retention Source does not set a value for this
property. However, I also noticed that if all the workunits have size 0 and
progress reporting is turned on then there will be a divide by 0 error
[here](https://jarvis.corp.linkedin.com/codesearch/result/?name=TaskStateCollectorService.java&path=gobblin-elr%2Fgobblin-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fruntime&reponame=linkedin%2Fgobblin-elr#281).
I defaulted to using "1" for the size so progress reporting will default to
being based on # work units. Although retention jobs will occur very quickly
the progress will appear "100%" instead of "0%" or "Not calculated" if I
handled the error within the TaskStateCollectorService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 650238)
Time Spent: 1h 10m (was: 1h)
> Set Default for Work Unit Size to Prevent Exception
> ---------------------------------------------------
>
> Key: GOBBLIN-1543
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1543
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Found error when summing work units for retention jobs that do not have size
> associated with it
>
> 02-09-2021 14:39:52 PDT gobblin-ivy_gobblin-ivy-download INFO - ERROR Failed
> to launch and run job
> job_ktwo_k2-cdp-company-features_DateTimeVerFinderBasedRetention_holdem_lasso_multi-hops-1-selfserve-timeaware-file-based-gen2-copy-azure_-809735922_1630618790955
> due tonull: java.lang.NumberFormatException: null
> ...
> org.apache.gobblin.runtime.AbstractJobLauncher.lambda$sumWorkUnitsSizes$1(AbstractJobLauncher.java:666)
> 02-09-2021 14:39:52 PDT gobblin-ivy_gobblin-ivy-download INFO - at
> org.apache.gobblin.runtime.AbstractJobLauncher.sumWorkUnitsSizes(AbstractJobLauncher.java:666)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
