[
https://issues.apache.org/jira/browse/GOBBLIN-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zihan Li updated GOBBLIN-1654:
------------------------------
Description: Now we calculate the capacity based on the consumer rate
during peak hour, but if there are records with super bad schema during that
time, our consumer rate will be super low and even after we catch up, we will
not be able to release the resources because of the low consumer rate. So want
provide a config to set the minimum consumer rate and avoid abusing resources
and small files (was: No we calculate the capacity based on the consumer rate
during peak hour, but if there is record with super bad schema during that
time, our consumer rate will be super low and even after we catch up, we will
not able to release the resources because of the low consumer rate. So want
provide a config to set the minimum consumer rate and avoid abusing resources
ad small files)
> Add capacity floor to avoid aggressively requesting resource and small files.
> -----------------------------------------------------------------------------
>
> Key: GOBBLIN-1654
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1654
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Zihan Li
> Priority: Major
>
> Now we calculate the capacity based on the consumer rate during peak hour,
> but if there are records with super bad schema during that time, our consumer
> rate will be super low and even after we catch up, we will not be able to
> release the resources because of the low consumer rate. So want provide a
> config to set the minimum consumer rate and avoid abusing resources and small
> files
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
