Kip Kohn created GOBBLIN-2179:
---------------------------------
Summary: Provide observability about how much work remains for a
job
Key: GOBBLIN-2179
URL: https://issues.apache.org/jira/browse/GOBBLIN-2179
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-core
Reporter: Kip Kohn
Assignee: Abhishek Tiwari
Provide observability about how much work remains for a job. In addition,
enable measurement and analysis about how long sub-tasks (`WorkUnit`s) take
while processing some amount of data.
We wish this info to feed into the analysis for Gobblin-on-Temporal (GoT)
auto-scaling and right-sizing, based on Dynamic Scaling -
https://issues.apache.org/jira/browse/GOBBLIN-2170
e.g. in `GenerateWorkUnits`' result, capture the total number of
`MultiWorkUnit`s and constituent sub-`WorkUnit`s, their total size in bytes, as
well as the size distribution across all WUs.
also, in the `WorkUnitClaimCheck` argument to each `ProcessWorkUnit` activity,
capture the total size of the possibly `MultiWorkUnit` as well as an
abbreviated characterization of the size of any constituent `WorkUnit`s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
