[
https://issues.apache.org/jira/browse/GOBBLIN-1703?focusedWorklogId=812301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812301
]
ASF GitHub Bot logged work on GOBBLIN-1703:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 27/Sep/22 00:30
Start Date: 27/Sep/22 00:30
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on code in PR #3550:
URL: https://github.com/apache/gobblin/pull/3550#discussion_r980612114
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlUserQuotaManager.java:
##########
@@ -62,7 +80,7 @@ public void init(Collection<Dag<JobExecutionPlan>> dags) {
@Override
int incrementJobCount(String user, CountType countType) throws IOException {
try {
- return this.mysqlStore.increaseCount(user, countType);
+ return this.quotaStore.increaseCount(user, countType);
Review Comment:
Question here, shouldn't we change the increaseCount and addDagId to be one
mysql transaction? Otherwise we will see discrepancy between these two table?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/MultiHopFlowCompiler.java:
##########
@@ -281,6 +281,13 @@ public Dag<JobExecutionPlan> compileFlow(Spec spec) {
Instrumented.markMeter(flowCompilationSuccessFulMeter);
Instrumented.updateTimer(flowCompilationTimer, System.nanoTime() -
startTime, TimeUnit.NANOSECONDS);
+ if
(Boolean.parseBoolean(flowSpec.getConfigAsProperties().getProperty(ServiceConfigKeys.GOBBLIN_SERVICE_ADHOC_FLOW)))
{
+ // todo : we should probably set it on all of the start nodes
+ for (Dag.DagNode<JobExecutionPlan> dagNode :
jobExecutionPlanDag.getStartNodes()) {
Review Comment:
If we already change it to start nodes, should we remove the todo log on
line 285 also change the logic in compiler onAddFlowSpec to check quota for all
start nodes as well?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/MultiHopFlowCompiler.java:
##########
@@ -281,6 +281,13 @@ public Dag<JobExecutionPlan> compileFlow(Spec spec) {
Instrumented.markMeter(flowCompilationSuccessFulMeter);
Instrumented.updateTimer(flowCompilationTimer, System.nanoTime() -
startTime, TimeUnit.NANOSECONDS);
+ if
(Boolean.parseBoolean(flowSpec.getConfigAsProperties().getProperty(ServiceConfigKeys.GOBBLIN_SERVICE_ADHOC_FLOW)))
{
+ // todo : we should probably set it on all of the start nodes
+ for (Dag.DagNode<JobExecutionPlan> dagNode :
jobExecutionPlanDag.getStartNodes()) {
Review Comment:
Also if we have running dag Ids, do we still need this? anyway we should be
able to avoid double count using that map?
Issue Time Tracking
-------------------
Worklog Id: (was: 812301)
Time Spent: 0.5h (was: 20m)
> avoid double quota usage increment for ad hoc flows
> ---------------------------------------------------
>
> Key: GOBBLIN-1703
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1703
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When a gaas flow request comes, resource handler checks the quota right there.
> However, if the flow has runImmediately=true, the quota will be checked again
> when the first job starts. This should be avoided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
