[
https://issues.apache.org/jira/browse/AIRFLOW-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341783#comment-15341783
]
Jeremiah Lowin commented on AIRFLOW-245:
----------------------------------------
The Scheduler is an object that decides what tasks to run; the Executor is an
object that decides where to run those tasks. So the Scheduler hands a
TaskInstance to the Executor and says "I want to run this task". That's why the
Executor has access to the TaskInstance when tasks are being queued. The
Executor then extracts the TI's "command" (for example: airflow run my_task)
and its sole job is to decide how to run that command. execute_async() is where
that decision is made. The important thing is that the thing being run is the
string command, not the TI object, and the signature of execute_async should
reflect that.
The larger question here is why would an operator ever influence the executor?
An Operator is a series of instructions that are designed to be run in a
self-contained way, no matter what the executor is. An executor is a way of
kicking off an Operator. There should be no dependency between them. To be
explicit: an Operator must be able to run under any Executor.
However, I am not familiar with Mesos so unfortunately I don't know exactly
what you're trying to do -- but have you looked at the existing MesosExecutor
in airflow.contrib.executors? Is it insufficient for your objectives? If your
operator requires special handling, then it sounds like maybe you want a
MesosOperator that runs your docker command?
> Access to task instance from custom Executor
> --------------------------------------------
>
> Key: AIRFLOW-245
> URL: https://issues.apache.org/jira/browse/AIRFLOW-245
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executor
> Reporter: Alexandr Nikitin
>
> I'm writing a custom executor that executes tasks on mesos and I want to have
> access to task instances from it. So that I can reuse all existing operators
> e.g. DockerOperator and access its fields like image, command, volumes and
> transform them to mesos.
> This can be done by changing `def execute_async(self, key, command,
> queue=None):` in `BaseExecutor`.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
