[
https://issues.apache.org/jira/browse/GOBBLIN-2186?focusedWorklogId=950622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-950622
]
ASF GitHub Bot logged work on GOBBLIN-2186:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 02/Jan/25 05:33
Start Date: 02/Jan/25 05:33
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #4089:
URL: https://github.com/apache/gobblin/pull/4089#discussion_r1900552161
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/activity/impl/GenerateWorkUnitsImpl.java:
##########
@@ -127,6 +130,9 @@ public GenerateWorkUnitsResult generateWorkUnits(Properties
jobProps, EventSubmi
int numSizeSummaryQuantiles =
getConfiguredNumSizeSummaryQuantiles(jobState);
WorkUnitsSizeSummary wuSizeSummary =
digestWorkUnitsSize(workUnits).asSizeSummary(numSizeSummaryQuantiles);
log.info("Discovered WorkUnits: {}", wuSizeSummary);
+ // IMPORTANT: send prior to `writeWorkUnits`, so the volume of work
discovered (and bin packed) gets durably measured. even if serialization were
to
+ // exceed available memory and this activity execution were to fail, a
subsequent re-attempt would know the amount of work, to guide re-config/attempt
+ createWorkPreparedSizeDistillationTimer(wuSizeSummary,
eventSubmitterContext).stop();
Review Comment:
having WU planning separate from WU serialization is worth considering.
that would allow for a re-attempt of only serialization w/o needing to rerun
the planning. there's no concern in sending the GTE again on the re-attempt -
that's not the motivation. instead the benefit would be to expedite the
re-attempt by subsequently skipping a repeat of successful WU planning.
to separate into two activities we'd need to succeed in persisting some
intermediate form of WU planning, so the WU planning activity could "pass
input" to the WU serialization activity, as the two won't execute together - or
possibly even on the same host. that intermediate form clearly ought to be
more likely to succeed than serializing all the WUs themselves - the failure
we're trying to address.
this major design choice awaits if we decide to pursue such larger rework.
Issue Time Tracking
-------------------
Worklog Id: (was: 950622)
Time Spent: 50m (was: 40m)
> Ensure GoT jobs record Work Discovery planning timing for populating the
> `GaaSJobObservabilityEvent` fields `jobPlanning{Start,End}Timestamp`
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-2186
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2186
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> `GaaSJobObservabilityEvent`s for Gobblin-on-Temporal jobs have no values set
> for the fields `jobPlanningStartTimestamp` and `jobPlanningEndTimestamp`
> because no `TimingEvent.LauncherTimings.WORK_UNITS_CREATION` GTE (to record
> those values) is emitted by `GenerateWorkUnitsImpl`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
