[
https://issues.apache.org/jira/browse/TEZ-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314960#comment-14314960
]
Siddharth Seth commented on TEZ-1233:
-------------------------------------
bq. Eg. If I have to take a job thats having issues and want to rerun it once
with a different config for a given vertex then can I do that after this change?
No, but that can be a follow up jira, if required. Like I mentioned in my
previous comment - I think defining something like this via configuration is
best left to frameworks making use of Tez. Programmatic configuration is still
helpful for case like #allowed attempts when a vertex may not be deterministic,
or for setting speculation per vertex (e.g. Pig determines a vertex to be a Map
/ Reduce and sets up speculation parameters to be based on MR parameters, which
from the looks of the PIG jira is what they'll do - at least initially).
bq. Thinking about the comments here and in TEZ-2003, I would expect that a
clear ExecutionContext like API would be created that would allow different
execution contexts to be specified per vertex in an unambiguous and clearly
defined manner. Instead it seems like the functionality depends on correctly
setting a bunch of independent configuration properties via a configuration
based API. Apart from being error prone, in general we have tried to stay clear
of configuration based API's in Tez. So the change concerns me.
For starters, this is going to be driven by programmatic configuration.
Ideally, there'll be a central context which can provide entities for the
various plugin points - TaskCommunication Plan, Scheduling, Launch. These will
be central entities (specified at DAG level) which can be shared across
multiple vertices. Individual vertices still need to specify what they intend
to use - which can be via configuration or again via an explicit API call. IAC,
that's a discussion for TEZ-2003, so I'll stop here.
> Allow configuration of framework parameters per vertex
> ------------------------------------------------------
>
> Key: TEZ-1233
> URL: https://issues.apache.org/jira/browse/TEZ-1233
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Attachments: TEZ-1233-1.patch, TEZ-1233-2.patch, TEZ-1233-3.patch,
> TEZ-1233-4.patch
>
>
> Currently, configuration properties specific in AMConfiguration are used to
> configure tasks - e.g. memory configs, process tree, etc. These should be
> configurable at a Vertex level.
> Also, tasks end up reading these configs from the dist cache each time -
> would be simpler to send over the wire as part of the TaskSpec.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
