[CONF] Apache Samza > SEP-22: Container Placements in Samza

[Sanil Jain (Confluence)]

Title: Message Title

There's 3 resolved inline comments and 1 new edit on this page

SEP-22: Container Placements in Samza 3 resolved inline comments Sanil Jain resolved individual namespaces Are requests and responses separate namespaces? If so, why? View comment Sanil Jain resolved processorId Where is UUID and deployment ID? In the payload?  Can you document the message structure (key and value). View comment Sanil Jain resolved delete messages Who does this? View comment 1 new edit Sanil Jain edited this page Here's what changed: ... Key for storing the ContainerPlacementRequestMessage & ContainerPlacementResponseMessage in Metastore is chosen to be UUID + "." + messageType(ContainerPlacementResponseMessage or ContainerPlacementRequestMessage). Value will be payload container ContainerPlacementRequestMessage & ContainerPlacementResponseMessage. Messages are written and read to the Metastore through the MetadataStore abstraction.  Since the metastore is eventually consistent, duplicate messages are required to be handled by ContainerPlacementService.  ContainerPlacementRequestMessage: Key Value Field Description Field Type "UUID.subType" uuid Unique identifier of a response message Required processorId  Logical processor id 0,1,2 of the container Required deploymentId Unique identifier for a deployment Required subType Type of message here: ContainerPlacementResponseMessage Required destinationHost Destination host where the container is desired to be moved Required statusCode Status of the current action Required responseMessage Response message in conjunction to status Required timestamp The timestamp of the response message Required requestExpiry Eequest expiry which acts as a timeout for any resource request to cluster resource manager Optional Sample KV Key Value [1,"samza-place-container-v1","f068175b-c9b6-4f34-982b-ecb5619f21de.ContainerPlacementRequestMessage"] {"processorId":"1","deploymentId":"app-atttempt-001","subType":"ContainerPlacementRequestMessage","uuid":"f068175b-c9b6-4f34-982b-ecb5619f21de","destinationHost":"ANY_HOST","statusCode":"CREATED","timestamp":1578693870484} ContainerPlacementResponseMessage: Key Value Field Description Field Type "UUID.subType" uuid Unique identifier of a response message Required processorId  Logical processor id 0,1,2 of the container Required deploymentId Unique identifier for a deployment Required subType Type of message here: ContainerPlacementResponseMessage Required destinationHost Destination host where the container is desired to be moved Required statusCode Status of the current action Required responseMessage Response message in conjunction to status Required timestamp The timestamp of the response message Required requestExpiry Eequest expiry which acts as a timeout for any resource request to cluster resource manager Optional Sample KV Key Value [1,"samza-place-container-v1","88b0d30c-d518-4307-9e8e-c8529eb30f04.ContainerPlacementResponseMessage"] {"processorId":"1","deploymentId":"app-atttempt-001","subType":"ContainerPlacementResponseMessage","responseMessage":"Request is accepted","uuid":"88b0d30c-d518-4307-9e8e-c8529eb30f04","destinationHost":"ANY_HOST","statusCode":"ACCEPTED","timestamp":1578694070875} GC policy for stale messages in metastore One way to delete stale ContainerPlacementMessages is to delete request / responses from the previous incarnation of the job in the metastore on job restarts Once the request is complete, ContainerPlacementService can issue an async delete to the metastore Request / response message can be externally cleaned by a tool Part 2. Container Placement Service ... Remove the HostAwareContainerAllocator & ContainerAllocator, simplify Container Allocator as a simple lightweight entity allocating requests to available resources (PR1, PR2) Introduce ContainerManager which acts as a brain for validating and issuing any actions on containers in the Job Coordinator for both active & Standby containers. (PR) Transfer state & validation of container launch & expired request handling from ContainerAllocator to ContainerManager Transfer state & validation for callback handler (ContainerProcessManager) to ContainerManager Encapsulates logic and state related to container placement actions like move, restarts for active & standby container in ContainerManager (PR-1, TDB) It is ContainerManager’s duty to validate any ContainerPlacementRequestMessages & also invalidate messages from the previous deployment incarnation It is ContainerManager’s duty to write ContainerPlacementResponseMessages to Metastore for external control to query the status of the request ContainerPlacementMetadata is a metadata holder for container actions (ControlActionMetadata) for ex request_id, current status, requested resources etc Image RemovedImage Added 2.1 Container Move 2.1.1 Stateless Container Move & Stateful Container Move (without Standby) ... Go to page history View page

Stop watching space • Manage notifications This message was sent by Atlassian Confluence 7.1.2