[
https://issues.apache.org/jira/browse/GOBBLIN-2085?focusedWorklogId=923168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923168
]
ASF GitHub Bot logged work on GOBBLIN-2085:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jun/24 18:55
Start Date: 12/Jun/24 18:55
Worklog Time Spent: 10m
Work Description: Will-Lo merged PR #3970:
URL: https://github.com/apache/gobblin/pull/3970
Issue Time Tracking
-------------------
Worklog Id: (was: 923168)
Time Spent: 0.5h (was: 20m)
> Increase `startToCloseTimeout` for `ExecuteGobblinWorkflow` activities
> ----------------------------------------------------------------------
>
> Key: GOBBLIN-2085
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2085
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> the currently hard-coded `startToCloseTimeout` values are too short.
> requirements that have come to light w/ operational experience:
> * `ProcessWorkUnits` must support extractors utilizing little parallelism
> (such as those reading from a DB) that take a very long time, even upwards of
> hours
> * `CommitActivity` runs may have to handle O(10k) or more task state files,
> which may take a long time to open and read, esp. when the `FileSystem` is
> under heavy load
> * `GenerateWorkUnits` must work with sources that may be quite vast, yet w/
> limited recourse to parallelism, such as a massive source iceberg
> ultimately these and other temporal config values would ideally be
> configurable, and that will come soon. for now, just bump to values large
> enough to resolve site-up issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
