azagrebin commented on a change in pull request #7822: [FLINK-11726][network]
Refactor the creation of ResultPartition and InputGate into NetworkEnvironment
URL: https://github.com/apache/flink/pull/7822#discussion_r267513101
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java
##########
@@ -202,28 +233,97 @@ public TaskKvStateRegistry
createKvStateTaskRegistry(JobID jobId, JobVertexID jo
return kvStateRegistry.createTaskRegistry(jobId, jobVertexId);
}
+ public ResultPartition[] getResultPartitionWriters(String taskId) {
+ return allPartitions.get(taskId);
+ }
+
+ public SingleInputGate[] getInputGates(String taskId) {
+ return allInputGates.get(taskId);
+ }
+
//
--------------------------------------------------------------------------------------------
- // Task operations
+ // Create Output Writers and Input Readers
//
--------------------------------------------------------------------------------------------
- public void registerTask(Task task) throws IOException {
- final ResultPartition[] producedPartitions =
task.getProducedPartitions();
+ public ResultPartitionWriter[] createResultPartitionWriters(
+ String taskId,
+ JobID jobId,
+ ExecutionAttemptID executionId,
+ TaskActions taskActions,
+ ResultPartitionConsumableNotifier
partitionConsumableNotifier,
+ TaskIOMetricGroup metrics,
+ Collection<ResultPartitionDeploymentDescriptor>
resultPartitionDeploymentDescriptors) throws IOException {
+
+ if (isShutdown) {
+ throw new IllegalStateException("NetworkEnvironment is
shut down");
+ }
- synchronized (lock) {
- if (isShutdown) {
- throw new
IllegalStateException("NetworkEnvironment is shut down");
- }
+ ResultPartition[] parititons = new
ResultPartition[resultPartitionDeploymentDescriptors.size()];
+ int counter = 0;
+ for (ResultPartitionDeploymentDescriptor rpdd :
resultPartitionDeploymentDescriptors) {
+ parititons[counter] = new ResultPartition(
+ taskId,
+ taskActions,
+ jobId,
+ new ResultPartitionID(rpdd.getPartitionId(),
executionId),
+ rpdd.getPartitionType(),
+ rpdd.getNumberOfSubpartitions(),
+ rpdd.getMaxParallelism(),
+ resultPartitionManager,
+ partitionConsumableNotifier,
+ ioManager,
+ rpdd.sendScheduleOrUpdateConsumersMessage());
+
+ setupPartition(parititons[counter]);
+ counter++;
+ }
- for (final ResultPartition partition :
producedPartitions) {
- setupPartition(partition);
+ allPartitions.put(taskId, parititons);
+
+ metrics.initializeOutputBufferMetrics(parititons);
Review comment:
Initially, we thought that ShuffleService would create separately
partitions/gates.
Now it seems that metrics and partition manager keep partitions/gates by
task.
It might change then ShuffleService interface.
As one of steps, we should probably refactor out netty specifics from
`TaskIOMetricGroup` and register those specific metrics inside netty
ShuffleService.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
