[
https://issues.apache.org/jira/browse/GOBBLIN-1656?focusedWorklogId=777877&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777877
]
ASF GitHub Bot logged work on GOBBLIN-1656:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 02/Jun/22 23:50
Start Date: 02/Jun/22 23:50
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on code in PR #3516:
URL: https://github.com/apache/gobblin/pull/3516#discussion_r888499897
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -322,6 +326,18 @@ public AddSpecResponse onAddSpec(Spec addedSpec) {
return new AddSpecResponse<>(response);
}
+ // Check quota limits against run immediately flows or adhoc flows before
saving the schedule
+ if (!jobConfig.containsKey(ConfigurationKeys.JOB_SCHEDULE_KEY) ||
PropertiesUtils.getPropAsBoolean(jobConfig,
ConfigurationKeys.FLOW_RUN_IMMEDIATELY, "false")) {
+ try {
+ if (quotaManager.isPresent()) {
+ quotaManager.get().checkQuota(dag.getNodes().get(0), false);
Review Comment:
Will this cause us to double the quota for this job since we check quota
again when submit the job?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceGuiceModule.java:
##########
@@ -194,6 +195,8 @@ public void configure(Binder binder) {
binder.bind(Orchestrator.class);
binder.bind(SchedulerService.class);
binder.bind(GobblinServiceJobScheduler.class);
+ OptionalBinder.newOptionalBinder(binder, UserQuotaManager.class);
Review Comment:
If this is needed?
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java:
##########
@@ -360,16 +361,21 @@ public Map<String, AddSpecResponse> put(Spec spec,
boolean triggerListener) {
responseMap.put(entry.getKey().getName(),
entry.getValue().getResult());
}
}
+ AddSpecResponse<String> schedulerResponse =
responseMap.getOrDefault(ServiceConfigKeys.GOBBLIN_SERVICE_JOB_SCHEDULER_LISTENER_CLASS,
new AddSpecResponse<>(null));
- if (isCompileSuccessful(responseMap)) {
+ if (isCompileSuccessful(schedulerResponse.getValue())) {
Review Comment:
When it exceed quota, why compile is marked as succeed?It's a little confuse
to read the code
Issue Time Tracking
-------------------
Worklog Id: (was: 777877)
Time Spent: 50m (was: 40m)
> Return different Http Status on GaaS if Quota is Exceeded
> ---------------------------------------------------------
>
> Key: GOBBLIN-1656
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1656
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> GaaS has quota limits for proxy users and flowGroups.
> When a user wants to create a flow that exceeds their specified quota, the
> flow should
> 1) Not be run
> 2) Return a http status code (i.e. 503) due to exceeding the resource. This
> allows clients to implement some wait and retry functionality
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
