zhuangchong commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r684865451
##########
File path:
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int
processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE,
DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE,
DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber ==
null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current
expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0
? average : average + 1;
+
+ Lists.partition(listDate,
slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE,
DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE,
DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
@kyoty
I feel that we are not talking about a part.
What you describe is the judgment operation of the first complement
generation task
What I am expressing is the execution operation after generating the
complement task,
In the complement parallel task, multiple serial tasks are generated
according to the custom parallelism. Each serial task has a start time and an
end time (the start time of the nth time). In this serial task, when the loop
is the nth time If the start time is equal to the end time, it will not be
executed this time.
My understanding may be wrong ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
