Albertus Kelvin created AIRFLOW-6249:
----------------------------------------

             Summary: Spark driver status polling for standalone, YARN, Mesos 
and K8s
                 Key: AIRFLOW-6249
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6249
             Project: Apache Airflow
          Issue Type: Improvement
          Components: hooks, operators
    Affects Versions: 1.10.6
            Reporter: Albertus Kelvin


I think the driver status tracker should also be enabled for yarn, mesos & 
kubernetes with cluster deploy mode.

Standalone, mesos & k8s cluster deploy mode: According to 
[this|[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L543]],
 *--status* parameter supports these cluster managers.

YARN cluster deploy mode: I think we can use built-in commands from yarn 
itself, such as *yarn application -status <ApplicationID>*.

Therefore, the *_build_track_driver_status_command* method should be updated 
accordingly to accommodate such a need, such as the following.
{code:python}
def _build_track_driver_status_command(self):
        # The driver id so we can poll for its status
        if not self._driver_id:
            raise AirflowException(
                    "Invalid status: attempted to poll driver " +
                    "status but no driver id is known. Giving up.")

        schemes = ("spark://", "mesos://", "k8s://https://";)
        if self._connection['master'].startswith(schemes): 
            # standalone, mesos, kubernetes
            connection_cmd = self._get_spark_binary_path()
            connection_cmd += ["--master", self._connection['master']]
            connection_cmd += ["--status", self._driver_id]
        else:
            # yarn
            connection_cmd = ["yarn application -status"]
            connection_cmd += [self._driver_id]

        self.log.debug("Poll driver status cmd: %s", connection_cmd)

        return connection_cmd{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to