[ https://issues.apache.org/jira/browse/AIRFLOW-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arthur Wiedmer resolved AIRFLOW-1028. ------------------------------------- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2202 [https://github.com/apache/incubator-airflow/pull/2202] > Databricks Operator for Airflow > ------------------------------- > > Key: AIRFLOW-1028 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1028 > Project: Apache Airflow > Issue Type: New Feature > Reporter: Andrew Chen > Assignee: Andrew Chen > Fix For: 1.9.0 > > > It would be nice to have a Databricks Operator/Hook in Airflow so users of > Databricks can more easily integrate with Airflow. > The operator would submit a spark job to our new /jobs/runs/submit endpoint. > This endpoint is similar to > https://docs.databricks.com/api/latest/jobs.html#jobscreatejob but does not > include the email_notifications, max_retries, min_retry_interval_millis, > retry_on_timeout, schedule, max_concurrent_runs fields. (The submit docs are > not out because it's still a private endpoint.) > Our proposed design for the operator then is to match this REST API endpoint. > Each argument to the parameter is named to be one of the fields of the REST > API request and the value of the argument will match the type expected by the > REST API. We will also merge extra keys from kwargs which should not be > passed to the BaseOperator into our API call in order to be flexible to > updates. > In the case that this interface is not very user friendly, we can later add > more operators which extend this operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)