[CONF] Apache Samza > SEP-23: Simplify Job Runner

[Ke Wu (Confluence)]

Title: Message Title

There's 13 inline comment updates and 2 new edits on this page

SEP-23: Simplify Job Runner 13 inline comment updates Ke Wu resolved pipeline difficult .. by introducing multiple points of failure. View comment Ke Wu resolved it requires isolation on the runner from security perspective to guard the framework from malicious user code Can you clarify what this means? Why is this different than any other process being executed for the job? View comment Ke Wu resolved In addition, config file is already packed in the tarball submitted to Yarn, it could be easier for AM to pick up the config locally. Let's clarify this. This is not a hard requirement. View comment Ke Wu Removed this part View comment • Like Ke Wu resolved and planning interface on AM Not pluggable planning, only config retrieval. View comment Ke Wu resolved when set ", which when used, will simplify the job submission to YARN." View comment Ke Wu resolved without involving any complex logic Redundant, can remove. View comment Ke Wu resolved provided config loader "using the provided config loader" View comment Ke Wu resolved persist back Persist what (clarify). View comment Ke Wu resolved For people who are interested in the new workflow To use the new workflow View comment Ke Wu resolved Two job configs "We will introduce." View comment Ke Wu resolved Fully backward compatible Incomplete sentence. View comment Ke Wu resolved Extra two configs in JobConfig we add to control "We will add two new configs to ..." View comment 2 new edits Ke Wu edited this page Here's what changed: ... Problem Samza Yarn follows a split multi-stage deployment model, where Job Runner, which runs on the submission host, reads configuration, performs planning and persist config in the coordinator stream before submitting the job to Yarn cluster. In Yarn, Application Master (AM) reads config from coordinator stream before spinning up containers to execute. Split of responsibility between job runner and AM is operationally confusing, and makes debugging the pipeline difficult with multiple points of failure. In addition, since planning invokes user code, it usually requires isolation on the runner from security perspective to guard the framework from malicious user code. In addition, config file is already packed in the tarball submitted to Yarn, it could be easier for AM to pick up the config locally., or a malicious user can gain access to other user jobs running on the same runner.  Proposed Changes We will provide a plugable config retrieval and planning interface retrieval interface on AM, when setused, Job Runner will simplify submit the job submission to Yarn, without involving any complex logic. AM on the other hand, will read job config by provided config using the provided config loader, performs planning, generate DAG and persist the final config back to coordinator stream. Public Interfaces Two We will introduce two job configs to configure the job to use the alternative workflow: job.config.loader.class job.config.loader.properties Fully The changes are fully backward compatible. For people who are interested in using the new workflow, simplify supply "job.config.loader.class" and "job.config.loader.properties". For example, in Hello Samza example, application will be invoked by ... Implementation and Test Plan JobConfig Extra We will add two new configs in JobConfig we add to control whether to read AM from ConfigLoader instead of coordinator stream. ... Go to page history View page

Stop watching space • Manage notifications This message was sent by Atlassian Confluence 6.15.8