[
https://issues.apache.org/jira/browse/GOBBLIN-2185?focusedWorklogId=949920&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-949920
]
ASF GitHub Bot logged work on GOBBLIN-2185:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Dec/24 04:05
Start Date: 24/Dec/24 04:05
Worklog Time Spent: 10m
Work Description: Blazer-007 commented on code in PR #4087:
URL: https://github.com/apache/gobblin/pull/4087#discussion_r1896349367
##########
gobblin-temporal/src/test/java/org/apache/gobblin/temporal/ddm/activity/impl/RecommendScalingForWorkUnitsLinearHeuristicImplTest.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.activity.impl;
+
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.testng.Assert;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import org.apache.gobblin.runtime.JobState;
+import org.apache.gobblin.temporal.GobblinTemporalConfigurationKeys;
+import org.apache.gobblin.temporal.ddm.work.TimeBudget;
+import org.apache.gobblin.temporal.ddm.work.WorkUnitsSizeSummary;
+
+
+/** Test for {@link RecommendScalingForWorkUnitsLinearHeuristicImpl} */
+public class RecommendScalingForWorkUnitsLinearHeuristicImplTest {
+
+ private RecommendScalingForWorkUnitsLinearHeuristicImpl scalingHeuristic;
+ @Mock private JobState jobState;
+ @Mock private WorkUnitsSizeSummary workUnitsSizeSummary;
+ @Mock private TimeBudget timeBudget;
+
+ @BeforeMethod
+ public void setUp() {
+ scalingHeuristic = new RecommendScalingForWorkUnitsLinearHeuristicImpl();
+ MockitoAnnotations.openMocks(this);
+ }
+
+ @Test
+ public void testCalcDerivationSetPoint() {
+
Mockito.when(jobState.getPropAsInt(Mockito.eq(GobblinTemporalConfigurationKeys.TEMPORAL_NUM_WORKERS_PER_CONTAINER),
Mockito.anyInt()))
+ .thenReturn(4); // 4 workers per container
+
Mockito.when(jobState.getPropAsLong(Mockito.eq(RecommendScalingForWorkUnitsLinearHeuristicImpl.AMORTIZED_NUM_BYTES_PER_MINUTE),
Mockito.anyLong()))
+ .thenReturn(100L * 1000 * 1000); // 100MB/minute
+ long targetTimeBudgetMinutes = 75L;
+
Mockito.when(timeBudget.getMaxTargetDurationMinutes()).thenReturn(targetTimeBudgetMinutes);
+
+ long totalNumMWUs = 3000L;
+
Mockito.when(workUnitsSizeSummary.getTopLevelWorkUnitsCount()).thenReturn(totalNumMWUs);
+
Mockito.when(workUnitsSizeSummary.getTopLevelWorkUnitsMeanSize()).thenReturn(500e6);
// 500MB
+ // parallelization capacity = 20 container-slots
+ // per-container-slot rate = 5 mins / MWU
Review Comment:
Can you please explain how this parallelization capacity &
per-container-slot rate is derived / calculated ?
Issue Time Tracking
-------------------
Worklog Id: (was: 949920)
Time Spent: 20m (was: 10m)
> Implement heuristic-based GoT Dynamic Auto-Scaling
> --------------------------------------------------
>
> Key: GOBBLIN-2185
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2185
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Using a configured (constant) Data Transfer Rate (in bytes per time), presume
> a linear relationship holds between "Work" (WU) throughput and scaling the
> number of worker-containers. Provide a heuristic-based recommendation for
> how many worker-containers to allocate in order to complete processing of a
> job within a given time budget, with volume of Work conveyed via
> `WorkUnitsSizeSummary`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
