[
https://issues.apache.org/jira/browse/TEZ-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Solal Pirelli updated TEZ-3841:
-------------------------------
Attachment: (was: TEZ-3841.patch)
> Proposal: Simulator mode
> ------------------------
>
> Key: TEZ-3841
> URL: https://issues.apache.org/jira/browse/TEZ-3841
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Solal Pirelli
> Attachments: TEZ-3841.patch
>
>
> Early work on a new feature proposal: a "simulator" mode in which vertices
> are not actually executed, but instead use a simplified "fake" processor
> (which is configurable, and by default does nothing) to let a developer see
> how certain workloads will be handled.
> For instance, one might want to check what happens if a vertex has each of
> its 1000 tasks send a bunch of events - does this scale? Or, what if a
> specific vertex fails 2% of the time - how does this impact overall graph
> execution? Are 2 nodes with 10 containers per node enough, or should one
> invest in a third node?
> My current implementation is pretty simple: mimic the "uber" stuff to add a
> new "fake" mode with a custom task scheduler and container launcher. It adds
> the following configuration values:
> * Boolean to enable fake mode
> * Number of nodes in fake mode
> * Number of containers per mode in fake mode
> * Class to run in fake mode - must inherit a new class `FakeProcessor`, with
> a single method `run` that takes the vertex name, task index and task
> attempt, and returns a list of events. Throwing an exception causes the task
> to fail.
> I'm currently working on adding a "chaos monkey" kind of service which
> randomly kills tasks, pre-empts containers, etc., but would appreciate some
> feedback on what's already done first. :)
> P.S.: I have zero experience with using JIRA or contributing to Apache
> projects; if there is a more formal procedure for suggesting a new feature,
> please point me to it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
