Kip Kohn created GOBBLIN-2085:
---------------------------------
Summary: Increase `startToCloseTimeout` for
`ExecuteGobblinWorkflow` activities
Key: GOBBLIN-2085
URL: https://issues.apache.org/jira/browse/GOBBLIN-2085
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-core
Reporter: Kip Kohn
Assignee: Abhishek Tiwari
the currently hard-coded `startToCloseTimeout` values are too short.
requirements that have come to light w/ operational experience:
* `ProcessWorkUnits` must support extractors utilizing little parallelism (such
as those reading from a DB) that take a very long time, even upwards of hours
* `CommitActivity` runs may have to handle O(10k) or more task state files,
which may take a long time to open and read, esp. when the `FileSystem` is
under heavy load
* `GenerateWorkUnits` must work with sources that may be quite vast, yet w/
limited recourse to parallelism, such as a massive source iceberg
ultimately these and other temporal config values would ideally be
configurable, and that will come soon. for now, just bump to values large
enough to resolve site-up issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
