lakshmi-manasa-g commented on a change in pull request #1448:
URL: https://github.com/apache/samza/pull/1448#discussion_r532984363
##########
File path:
samza-core/src/main/java/org/apache/samza/clustermanager/SamzaApplicationState.java
##########
@@ -105,6 +105,22 @@
*/
public final ConcurrentMap<String, SamzaResource> runningProcessors = new
ConcurrentHashMap<>(0);
+ /**
+ * Map of Samza processor Id (aka logical id) to execution environment
container id (aka physical id ex: yarn container id).
+ * This map will be used during the start up phase of new AM in AM-HA.
+ *
+ * This map is populated at startup of ClusterBasedJobCoordinator.
+ * It initially holds the processId to execution id mapping (if any) present
in the coordinator stream.
+ * This could correspond to containers currently running or from previous
attempt or previous deploy.
+ *
+ * If # of containers in map is same as current JobModel's containers, and
mapping is from previous deploy,
+ * then they will get overwritten by new container incarnations in the
current deploy in {@link
ContainerProcessManager}.onStreamProcessorLaunchSuccess.
+ * If # of containers in map is lesser, then map entries will get
overwritten by current containers and new ones will be added to map.
+ * If # of containers in map is greater, this map is wiped clear in {@link
ContainerProcessManager}.start
+ * to avoid having mapping for containers which are not part of the current
JobModel.
+ */
+ public final ConcurrentMap<String, String> processorToExecutionId = new
ConcurrentHashMap<>(0);
Review comment:
agreed.
I did consider not persisting this mapping in the state throughout lifecycle
of the AM but decided to go with it as other option of passing mapping/metadata
store to the life cycle will require changing some interfaces.
However, after our offline sync it does sound better to update the interface
than persist this map.
created a jira to follow it up SAMZA-2607.
also removed the updates to this mapping in the state to keep in fresh in CPM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
