dnishimura commented on issue #1079: SAMZA-2250: Support large job models in standalone. URL: https://github.com/apache/samza/pull/1079#issuecomment-502163083 Nice summary @shanthoosh. Few questions below > > * Number of `MetadataStore` writes: Current job model storage mechanism used for YARN is write intensive. JobModel is composed of different kind of metadata viz `TaskPartitionAssignment`, `TaskChangeLogAssignment`, `TaskContainerAssignment`, `Locality` etc and they are stored at task granularity independently. For a samza job that consumes from N input topic partitions, employing `YARN` metadata-storage-managers would result in 4 * N writes to the `MetadataStore` layer(For both `CoordinatorStreamStore`/`ZkMetadataStore`). Even after tuning kafka batch size, this is expensive for large standalone jobs(Example: Search standalone jobs consumes from 15K topic partitions on average). When you mention "tuning kafka batch size" do you mean batch puts? (i.e. flush after M entries). Correct me if I'm wrong, but isn't the performance hit mainly because of the current way of "flush per entry"? Do we know if batch puts would increase performance by an order of magnitude or more or would the increase in performance be much less? > * JobModel write frequency: Currently in YARN, the JobModel is computed and stored in `MetadataStore` only once in `ApplicationMaster` startup. In standalone, JobModel can be computed and published multiple times as a part of a single deployment. This depends on the number of hosts and the configured debounce time? > > B. _Versioning:_ JobModel versioning is not supported in coordinator stream currently. Let's say leader dies after publishing some portion of JobModel. From the coordinator stream alone, we cannot correctly compute the correct JobModel in standalone currently. This is required for jobmodel-lineage tracking and versioned JobModel read for functional correctness. Seems like ZK is the only option then. > > C. _Observability:_ Currently YARN has dashboard support for viewing the JobModel. If we use coordinator stream to store JobModel in standalone then there's no tooling around JobModel visibility. This will make trouble shooting a standalone job harder. If we create a metadata store tool, this problem should be solved.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
