[jira] [Work started] (AIRFLOW-2493) Mark template_fields of all Operators in the API document as "templated"
[ https://issues.apache.org/jira/browse/AIRFLOW-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-2493 started by roc chan. - > Mark template_fields of all Operators in the API document as "templated" > > > Key: AIRFLOW-2493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2493 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Minor > > I saw some words from Concepts-Additional Functionality-Jinjia Templating in > the document : > [http://airflow.readthedocs.io/en/latest/concepts.html?highlight=xcom#jinja-templating] > > *You can use Jinja templating with every parameter that is marked as > “templated” in the documentation. Template substitution occurs just before > the pre_execute function of your operator is called.* > But the actual situation of the API document is not like this，only a small > portion of template_fields have marked "(templated)", for example, > *BashOperator*'s *env* has been marked, but *bash_command* is not > ([http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)|http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)*,*], > which has caused some confusion. I want every template_fields of all > Operators in the API document has marked "(templated)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2493) Mark template_fields of all Operators in the API document as "templated"
[ https://issues.apache.org/jira/browse/AIRFLOW-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481878#comment-16481878 ] roc chan commented on AIRFLOW-2493: --- https://github.com/apache/incubator-airflow/pull/3386 > Mark template_fields of all Operators in the API document as "templated" > > > Key: AIRFLOW-2493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2493 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Minor > > I saw some words from Concepts-Additional Functionality-Jinjia Templating in > the document : > [http://airflow.readthedocs.io/en/latest/concepts.html?highlight=xcom#jinja-templating] > > *You can use Jinja templating with every parameter that is marked as > “templated” in the documentation. Template substitution occurs just before > the pre_execute function of your operator is called.* > But the actual situation of the API document is not like this，only a small > portion of template_fields have marked "(templated)", for example, > *BashOperator*'s *env* has been marked, but *bash_command* is not > ([http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)|http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)*,*], > which has caused some confusion. I want every template_fields of all > Operators in the API document has marked "(templated)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2494) Bump FlaskAppBuilder to 1.11.1
Fokko Driesprong created AIRFLOW-2494: - Summary: Bump FlaskAppBuilder to 1.11.1 Key: AIRFLOW-2494 URL: https://issues.apache.org/jira/browse/AIRFLOW-2494 Project: Apache Airflow Issue Type: Improvement Reporter: Fokko Driesprong Assignee: Fokko Driesprong Which allows us to bump Flask to 0.12.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1
Repository: incubator-airflow Updated Branches: refs/heads/master 67b351183 -> b755d3547 [AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1 This will bring Airflow back up to date and will allow us to run Flask 0.12.4 Closes #3382 from Fokko/AIRFLOW-2489-update- dependencies Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b755d354 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b755d354 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b755d354 Branch: refs/heads/master Commit: b755d35479ecd77bc264471aa830a90d5ca66211 Parents: 67b3511 Author: Fokko Driesprong Authored: Sun May 20 12:00:52 2018 +0100 Committer: Kaxil Naik Committed: Sun May 20 12:00:52 2018 +0100 -- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b755d354/setup.py -- diff --git a/setup.py b/setup.py index 575ea0d..299a1db 100644 --- a/setup.py +++ b/setup.py @@ -248,8 +248,8 @@ def do_setup(): 'configparser>=3.5.0, <3.6.0', 'croniter>=0.3.17, <0.4', 'dill>=0.2.2, <0.3', -'flask>=0.12, <0.12.2', -'flask-appbuilder>=1.10.0, <2.0.0', +'flask>=0.12.4, <0.13', +'flask-appbuilder>=1.11.1, <2.0.0', 'flask-admin==1.4.1', 'flask-caching>=1.3.3, <1.4.0', 'flask-login==0.2.11',
[jira] [Commented] (AIRFLOW-2489) Align Flask dependencies with FlaskAppBuilder
[
https://issues.apache.org/jira/browse/AIRFLOW-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481930#comment-16481930
]
ASF subversion and git services commented on AIRFLOW-2489:
--
Commit b755d35479ecd77bc264471aa830a90d5ca66211 in incubator-airflow's branch
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b755d35 ]
[AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1
This will bring Airflow back up to date and will
allow us to run
Flask 0.12.4
Closes #3382 from Fokko/AIRFLOW-2489-update-
dependencies
> Align Flask dependencies with FlaskAppBuilder
> -
>
> Key: AIRFLOW-2489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2489
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
> Right now it might take a while to update the dependencies. And we would like
> to update the dependencies to make sure that we don't have any version
> conflicts like:
> Traceback (most recent call last):
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/bin/airflow",
> line 4, in
>
> __import__('pkg_resources').require('apache-airflow==2.0.0.dev0+incubating')
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3086, in
> @_call_aside
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3070, in _call_aside
> f(*args, **kwargs)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3099, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 576, in _build_master
> return cls._build_from_requirements(__requires__)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 589, in _build_from_requirements
> dists = ws.resolve(reqs, Environment())
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 783, in resolve
> raise VersionConflict(dist, req).with_context(dependent_req)
> pkg_resources.ContextualVersionConflict: (Flask 0.12.4
> (/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages),
> Requirement.parse('Flask<0.12.2,>=0.10.0'), set(['flask-appbuilder']))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2489) Align Flask dependencies with FlaskAppBuilder
[
https://issues.apache.org/jira/browse/AIRFLOW-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481931#comment-16481931
]
ASF subversion and git services commented on AIRFLOW-2489:
--
Commit b755d35479ecd77bc264471aa830a90d5ca66211 in incubator-airflow's branch
refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b755d35 ]
[AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1
This will bring Airflow back up to date and will
allow us to run
Flask 0.12.4
Closes #3382 from Fokko/AIRFLOW-2489-update-
dependencies
> Align Flask dependencies with FlaskAppBuilder
> -
>
> Key: AIRFLOW-2489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2489
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
> Right now it might take a while to update the dependencies. And we would like
> to update the dependencies to make sure that we don't have any version
> conflicts like:
> Traceback (most recent call last):
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/bin/airflow",
> line 4, in
>
> __import__('pkg_resources').require('apache-airflow==2.0.0.dev0+incubating')
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3086, in
> @_call_aside
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3070, in _call_aside
> f(*args, **kwargs)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3099, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 576, in _build_master
> return cls._build_from_requirements(__requires__)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 589, in _build_from_requirements
> dists = ws.resolve(reqs, Environment())
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 783, in resolve
> raise VersionConflict(dist, req).with_context(dependent_req)
> pkg_resources.ContextualVersionConflict: (Flask 0.12.4
> (/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages),
> Requirement.parse('Flask<0.12.2,>=0.10.0'), set(['flask-appbuilder']))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2489) Align Flask dependencies with FlaskAppBuilder
[
https://issues.apache.org/jira/browse/AIRFLOW-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2489.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Resolved by https://github.com/apache/incubator-airflow/pull/3382
Commit:
https://github.com/apache/incubator-airflow/commit/b755d35479ecd77bc264471aa830a90d5ca66211
> Align Flask dependencies with FlaskAppBuilder
> -
>
> Key: AIRFLOW-2489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2489
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>
> Right now it might take a while to update the dependencies. And we would like
> to update the dependencies to make sure that we don't have any version
> conflicts like:
> Traceback (most recent call last):
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/bin/airflow",
> line 4, in
>
> __import__('pkg_resources').require('apache-airflow==2.0.0.dev0+incubating')
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3086, in
> @_call_aside
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3070, in _call_aside
> f(*args, **kwargs)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 3099, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 576, in _build_master
> return cls._build_from_requirements(__requires__)
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 589, in _build_from_requirements
> dists = ws.resolve(reqs, Environment())
> File
> "/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 783, in resolve
> raise VersionConflict(dist, req).with_context(dependent_req)
> pkg_resources.ContextualVersionConflict: (Flask 0.12.4
> (/home/travis/build/Fokko/incubator-airflow/.tox/py27-backend_postgres/lib/python2.7/site-packages),
> Requirement.parse('Flask<0.12.2,>=0.10.0'), set(['flask-appbuilder']))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[1/2] incubator-airflow git commit: [AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated"
Repository: incubator-airflow Updated Branches: refs/heads/master b755d3547 -> dc78b9196 http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/mysql_operator.py -- diff --git a/airflow/operators/mysql_operator.py b/airflow/operators/mysql_operator.py index e3f1cd7..f7c94bf 100644 --- a/airflow/operators/mysql_operator.py +++ b/airflow/operators/mysql_operator.py @@ -27,7 +27,7 @@ class MySqlOperator(BaseOperator): :param mysql_conn_id: reference to a specific mysql database :type mysql_conn_id: string -:param sql: the sql code to be executed +:param sql: the sql code to be executed. (templated) :type sql: Can receive a str representing a sql statement, a list of str (sql statements), or reference to a template file. Template reference are recognized by str ending in '.sql' http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/mysql_to_hive.py -- diff --git a/airflow/operators/mysql_to_hive.py b/airflow/operators/mysql_to_hive.py index 2915b7e..22b7ac2 100644 --- a/airflow/operators/mysql_to_hive.py +++ b/airflow/operators/mysql_to_hive.py @@ -43,10 +43,10 @@ class MySqlToHiveTransfer(BaseOperator): stage the data into a temporary table before loading it into its final destination using a ``HiveOperator``. -:param sql: SQL query to execute against the MySQL database +:param sql: SQL query to execute against the MySQL database. (templated) :type sql: str :param hive_table: target Hive table, use dot notation to target a -specific database +specific database. (templated) :type hive_table: str :param create: whether to create the table if it doesn't exist :type create: bool @@ -54,7 +54,7 @@ class MySqlToHiveTransfer(BaseOperator): execution :type recreate: bool :param partition: target partition as a dict of partition columns -and values +and values. (templated) :type partition: dict :param delimiter: field delimiter in the file :type delimiter: str http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/oracle_operator.py -- diff --git a/airflow/operators/oracle_operator.py b/airflow/operators/oracle_operator.py index 1ab44a4..c08908b 100644 --- a/airflow/operators/oracle_operator.py +++ b/airflow/operators/oracle_operator.py @@ -26,7 +26,7 @@ class OracleOperator(BaseOperator): Executes sql code in a specific Oracle database :param oracle_conn_id: reference to a specific Oracle database :type oracle_conn_id: string -:param sql: the sql code to be executed +:param sql: the sql code to be executed. (templated) :type sql: Can receive a str representing a sql statement, a list of str (sql statements), or reference to a template file. Template reference are recognized by str ending in '.sql' http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/pig_operator.py -- diff --git a/airflow/operators/pig_operator.py b/airflow/operators/pig_operator.py index 05cc565..f7babcf 100644 --- a/airflow/operators/pig_operator.py +++ b/airflow/operators/pig_operator.py @@ -27,7 +27,7 @@ class PigOperator(BaseOperator): """ Executes pig script. -:param pig: the pig latin script to be executed +:param pig: the pig latin script to be executed. (templated) :type pig: string :param pig_cli_conn_id: reference to the Hive database :type pig_cli_conn_id: string http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/postgres_operator.py -- diff --git a/airflow/operators/postgres_operator.py b/airflow/operators/postgres_operator.py index aff3e69..5935f83 100644 --- a/airflow/operators/postgres_operator.py +++ b/airflow/operators/postgres_operator.py @@ -27,7 +27,7 @@ class PostgresOperator(BaseOperator): :param postgres_conn_id: reference to a specific postgres database :type postgres_conn_id: string -:param sql: the sql code to be executed +:param sql: the sql code to be executed. (templated) :type sql: Can receive a str representing a sql statement, a list of str (sql statements), or reference to a template file. Template reference are recognized by str ending in '.sql' http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/operators/presto_to_mysql.py -- diff --git a/airflow/operators/presto_to_mysql.py b/airflow/operators/presto_to_mysql.py index a4abc1f.
[jira] [Commented] (AIRFLOW-2493) Mark template_fields of all Operators in the API document as "templated"
[ https://issues.apache.org/jira/browse/AIRFLOW-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481953#comment-16481953 ] ASF subversion and git services commented on AIRFLOW-2493: -- Commit dc78b9196723ca6724185231ccd6f5bbe8edcaf3 in incubator-airflow's branch refs/heads/master from roc [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dc78b91 ] [AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated" Make all the "template_fields" (jinjia template) of all Operators marked as "templated" in the API document. Closes #3386 from imroc/AIRFLOW-2493 > Mark template_fields of all Operators in the API document as "templated" > > > Key: AIRFLOW-2493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2493 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Minor > > I saw some words from Concepts-Additional Functionality-Jinjia Templating in > the document : > [http://airflow.readthedocs.io/en/latest/concepts.html?highlight=xcom#jinja-templating] > > *You can use Jinja templating with every parameter that is marked as > “templated” in the documentation. Template substitution occurs just before > the pre_execute function of your operator is called.* > But the actual situation of the API document is not like this，only a small > portion of template_fields have marked "(templated)", for example, > *BashOperator*'s *env* has been marked, but *bash_command* is not > ([http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)|http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)*,*], > which has caused some confusion. I want every template_fields of all > Operators in the API document has marked "(templated)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[2/2] incubator-airflow git commit: [AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated"
[AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated" Make all the "template_fields" (jinjia template) of all Operators marked as "templated" in the API document. Closes #3386 from imroc/AIRFLOW-2493 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/dc78b919 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/dc78b919 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/dc78b919 Branch: refs/heads/master Commit: dc78b9196723ca6724185231ccd6f5bbe8edcaf3 Parents: b755d35 Author: roc Authored: Sun May 20 14:03:58 2018 +0100 Committer: Kaxil Naik Committed: Sun May 20 14:03:58 2018 +0100 -- airflow/contrib/operators/awsbatch_operator.py | 3 +- airflow/contrib/operators/bigquery_get_data.py | 6 +- airflow/contrib/operators/bigquery_operator.py | 24 +++--- .../operators/bigquery_table_delete_operator.py | 2 +- .../contrib/operators/bigquery_to_bigquery.py | 7 +- airflow/contrib/operators/bigquery_to_gcs.py| 6 +- .../contrib/operators/databricks_operator.py| 3 +- airflow/contrib/operators/dataproc_operator.py | 82 +++- .../operators/discord_webhook_operator.py | 4 +- airflow/contrib/operators/ecs_operator.py | 2 +- .../contrib/operators/emr_add_steps_operator.py | 4 +- .../operators/emr_create_job_flow_operator.py | 3 +- .../emr_terminate_job_flow_operator.py | 2 +- airflow/contrib/operators/file_to_gcs.py| 6 +- airflow/contrib/operators/file_to_wasb.py | 6 +- .../contrib/operators/gcs_download_operator.py | 8 +- airflow/contrib/operators/gcs_list_operator.py | 7 +- airflow/contrib/operators/gcs_operator.py | 8 +- airflow/contrib/operators/gcs_to_bq.py | 10 +-- airflow/contrib/operators/gcs_to_gcs.py | 10 +-- airflow/contrib/operators/gcs_to_s3.py | 9 ++- airflow/contrib/operators/hipchat_operator.py | 4 +- airflow/contrib/operators/hive_to_dynamodb.py | 2 +- .../operators/jenkins_job_trigger_operator.py | 2 +- airflow/contrib/operators/jira_operator.py | 2 +- .../operators/kubernetes_pod_operator.py| 6 +- airflow/contrib/operators/mlengine_operator.py | 49 ++-- airflow/contrib/operators/pubsub_operator.py| 3 +- airflow/contrib/operators/s3_list_operator.py | 6 +- airflow/contrib/operators/s3_to_gcs_operator.py | 8 +- .../operators/segment_track_event_operator.py | 6 +- airflow/contrib/operators/sftp_operator.py | 4 +- airflow/contrib/operators/snowflake_operator.py | 2 +- airflow/contrib/operators/spark_sql_operator.py | 2 +- .../contrib/operators/spark_submit_operator.py | 7 +- airflow/contrib/operators/ssh_operator.py | 2 +- airflow/contrib/operators/vertica_operator.py | 2 +- airflow/contrib/operators/vertica_to_hive.py| 7 +- airflow/contrib/operators/vertica_to_mysql.py | 8 +- airflow/contrib/operators/winrm_operator.py | 2 +- airflow/operators/bash_operator.py | 2 +- airflow/operators/check_operator.py | 4 +- airflow/operators/docker_operator.py| 4 +- airflow/operators/email_operator.py | 8 +- airflow/operators/generic_transfer.py | 6 +- airflow/operators/hive_operator.py | 7 +- airflow/operators/hive_stats_operator.py| 4 +- airflow/operators/hive_to_druid.py | 6 +- airflow/operators/hive_to_mysql.py | 14 ++-- airflow/operators/hive_to_samba_operator.py | 2 +- airflow/operators/http_operator.py | 4 +- airflow/operators/jdbc_operator.py | 2 +- airflow/operators/mssql_operator.py | 3 +- airflow/operators/mssql_to_hive.py | 9 ++- airflow/operators/mysql_operator.py | 2 +- airflow/operators/mysql_to_hive.py | 6 +- airflow/operators/oracle_operator.py| 2 +- airflow/operators/pig_operator.py | 2 +- airflow/operators/postgres_operator.py | 2 +- airflow/operators/presto_to_mysql.py| 10 +-- airflow/operators/python_operator.py| 2 +- airflow/operators/s3_file_transform_operator.py | 4 +- airflow/operators/s3_to_hive_operator.py| 6 +- airflow/operators/slack_operator.py | 10 ++- airflow/operators/sqlite_operator.py| 2 +- 65 files changed, 240 insertions(+), 219 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dc78b919/airflow/contrib/operators/awsbatch_operator.py -- diff --git a/airflow/contrib/operators/awsbatch_operator.py b/airflow/contrib/operators/awsbatch_operator.p
[jira] [Commented] (AIRFLOW-2493) Mark template_fields of all Operators in the API document as "templated"
[ https://issues.apache.org/jira/browse/AIRFLOW-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481952#comment-16481952 ] ASF subversion and git services commented on AIRFLOW-2493: -- Commit dc78b9196723ca6724185231ccd6f5bbe8edcaf3 in incubator-airflow's branch refs/heads/master from roc [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dc78b91 ] [AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated" Make all the "template_fields" (jinjia template) of all Operators marked as "templated" in the API document. Closes #3386 from imroc/AIRFLOW-2493 > Mark template_fields of all Operators in the API document as "templated" > > > Key: AIRFLOW-2493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2493 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Minor > > I saw some words from Concepts-Additional Functionality-Jinjia Templating in > the document : > [http://airflow.readthedocs.io/en/latest/concepts.html?highlight=xcom#jinja-templating] > > *You can use Jinja templating with every parameter that is marked as > “templated” in the documentation. Template substitution occurs just before > the pre_execute function of your operator is called.* > But the actual situation of the API document is not like this，only a small > portion of template_fields have marked "(templated)", for example, > *BashOperator*'s *env* has been marked, but *bash_command* is not > ([http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)|http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)*,*], > which has caused some confusion. I want every template_fields of all > Operators in the API document has marked "(templated)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2493) Mark template_fields of all Operators in the API document as "templated"
[ https://issues.apache.org/jira/browse/AIRFLOW-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2493. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/3386 Commit: https://github.com/apache/incubator-airflow/commit/dc78b9196723ca6724185231ccd6f5bbe8edcaf3 > Mark template_fields of all Operators in the API document as "templated" > > > Key: AIRFLOW-2493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2493 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Minor > Fix For: 2.0.0 > > > I saw some words from Concepts-Additional Functionality-Jinjia Templating in > the document : > [http://airflow.readthedocs.io/en/latest/concepts.html?highlight=xcom#jinja-templating] > > *You can use Jinja templating with every parameter that is marked as > “templated” in the documentation. Template substitution occurs just before > the pre_execute function of your operator is called.* > But the actual situation of the API document is not like this，only a small > portion of template_fields have marked "(templated)", for example, > *BashOperator*'s *env* has been marked, but *bash_command* is not > ([http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)|http://airflow.readthedocs.io/en/latest/code.html#airflow.operators.bash_operator.BashOperator)*,*], > which has caused some confusion. I want every template_fields of all > Operators in the API document has marked "(templated)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2494) Bump FlaskAppBuilder to 1.11.1
[ https://issues.apache.org/jira/browse/AIRFLOW-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2494. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/3382 Commit: https://github.com/apache/incubator-airflow/commit/b755d35479ecd77bc264471aa830a90d5ca66211 > Bump FlaskAppBuilder to 1.11.1 > -- > > Key: AIRFLOW-2494 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2494 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > Which allows us to bump Flask to 0.12.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2479) Improve doc FAQ section
[ https://issues.apache.org/jira/browse/AIRFLOW-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2479: Description: Wrongly named Jira Summary: Improve doc FAQ section (was: Create doc for RBAC UI) > Improve doc FAQ section > --- > > Key: AIRFLOW-2479 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2479 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.0, 2.0.0 > > > Wrongly named Jira -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2478) Improve doc FAQ section
[ https://issues.apache.org/jira/browse/AIRFLOW-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-2478. --- Resolution: Duplicate > Improve doc FAQ section > --- > > Key: AIRFLOW-2478 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2478 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2478) Improve doc FAQ section
[ https://issues.apache.org/jira/browse/AIRFLOW-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481957#comment-16481957 ] Kaxil Naik commented on AIRFLOW-2478: - Duplicate of https://issues.apache.org/jira/browse/AIRFLOW-2479 > Improve doc FAQ section > --- > > Key: AIRFLOW-2478 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2478 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2443) Fix doc links to pythonhosted
[ https://issues.apache.org/jira/browse/AIRFLOW-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-2443. --- Resolution: Fixed Fix Version/s: (was: Airflow 1.9.0) 1.10.0 This should be resolved in 1.10 release > Fix doc links to pythonhosted > - > > Key: AIRFLOW-2443 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2443 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.9.0 >Reporter: Anand >Priority: Minor > Fix For: 1.10.0 > > > Airflow 1.9 UI still redirects to incorrect documentation link on > pythonhosted.org, though seems to have been fixed in master in this bug - > https://issues.apache.org/jira/browse/AIRFLOW-2115 . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2443) Fix doc links to pythonhosted
[ https://issues.apache.org/jira/browse/AIRFLOW-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481960#comment-16481960 ] Kaxil Naik commented on AIRFLOW-2443: - Resolved by https://issues.apache.org/jira/browse/AIRFLOW-2115 . But will be seen in 1.10.0 > Fix doc links to pythonhosted > - > > Key: AIRFLOW-2443 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2443 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.9.0 >Reporter: Anand >Priority: Minor > Fix For: 1.10.0 > > > Airflow 1.9 UI still redirects to incorrect documentation link on > pythonhosted.org, though seems to have been fixed in master in this bug - > https://issues.apache.org/jira/browse/AIRFLOW-2115 . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2495) Celery should depend on kombu >= 4.1.0 for sqla transport
Craig Rodrigues created AIRFLOW-2495:
Summary: Celery should depend on kombu >= 4.1.0 for sqla transport
Key: AIRFLOW-2495
URL: https://issues.apache.org/jira/browse/AIRFLOW-2495
Project: Apache Airflow
Issue Type: Improvement
Components: celery
Affects Versions: 1.10
Reporter: Craig Rodrigues
I created a small requirements.txt that has one line:
{{git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]}}
I created a virtualenv, and did:
{{pip install -r requrements.txt}}
During the install I saw this:
{{Collecting kombu<5.0,>=4.0.2 (from
celery>=4.0.2->apache-airflow[celery,emr,hdfs,hive,ldap,mysql,postgres,redis,s3,slack]->-r
requirements.txt (line 1))}}
I had a system where kombu 4.0.2 was being installed.
This is bad because I also had in my airflow.cfg:
{{broker_url = sqla+mysql://airflow:password@hostname:3306/database}}
My celery transport was not working.
kombu 4.0.2 does not support sqlalchemy urls:
[https://github.com/celery/kombu/blob/v4.0.2/kombu/transport/__init__.py#L20]
but kombu 4.1.0 does:
[https://github.com/celery/kombu/blob/v4.1.0/kombu/transport/__init__.py#L32]
This is also mentioned in:
https://github.com/celery/kombu/issues/686
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2495) Celery should depend on kombu >= 4.1.0 for sqla transport
[
https://issues.apache.org/jira/browse/AIRFLOW-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482070#comment-16482070
]
Craig Rodrigues commented on AIRFLOW-2495:
--
See:
https://github.com/apache/incubator-airflow/pull/3388
> Celery should depend on kombu >= 4.1.0 for sqla transport
> -
>
> Key: AIRFLOW-2495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2495
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery
>Affects Versions: 1.10
>Reporter: Craig Rodrigues
>Priority: Major
>
> I created a small requirements.txt that has one line:
>
> {{git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]}}
>
> I created a virtualenv, and did:
>
> {{pip install -r requrements.txt}}
>
> During the install I saw this:
>
> {{Collecting kombu<5.0,>=4.0.2 (from
> celery>=4.0.2->apache-airflow[celery,emr,hdfs,hive,ldap,mysql,postgres,redis,s3,slack]->-r
> requirements.txt (line 1))}}
>
> I had a system where kombu 4.0.2 was being installed.
> This is bad because I also had in my airflow.cfg:
> {{broker_url = sqla+mysql://airflow:password@hostname:3306/database}}
>
> My celery transport was not working.
>
>
> kombu 4.0.2 does not support sqlalchemy urls:
> [https://github.com/celery/kombu/blob/v4.0.2/kombu/transport/__init__.py#L20]
> but kombu 4.1.0 does:
> [https://github.com/celery/kombu/blob/v4.1.0/kombu/transport/__init__.py#L32]
>
> This is also mentioned in:
> https://github.com/celery/kombu/issues/686
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2496) v1-10-test branch reports version 2.0 instead of 1.10
Craig Rodrigues created AIRFLOW-2496: Summary: v1-10-test branch reports version 2.0 instead of 1.10 Key: AIRFLOW-2496 URL: https://issues.apache.org/jira/browse/AIRFLOW-2496 Project: Apache Airflow Issue Type: Bug Components: release Affects Versions: 1.10 Reporter: Craig Rodrigues Fix For: 1.10 I created a requirements.txt with one line: git+ [https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]] I then did: 1. create a virtual environment 2. pip install -r requirements.txt 3. airflow webserver When I look at the version in the web interface, it shows a version of: 2.0.0.dev0+incubating even though I used the v1-10-test branch. This seems wrong. It ooks like these two commits got merged to v1-10-test branch which bump the version to 2.0: [https://github.com/apache/incubator-airflow/commit/305a787] [https://github.com/apache/incubator-airflow/commit/a30acaf] That seems wrong for v1-10-test branch. It would be nice if this version was 1.10.0.dev0+incubating (or whatever), since it looks like I will need to deploy v1-10-test branch to prod this week, and then very soon after when 1.10 is released, re-deploy airflow 1.10. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2497) Cgroup task runner doesn't pass down correct env vars
Kevin Yang created AIRFLOW-2497: --- Summary: Cgroup task runner doesn't pass down correct env vars Key: AIRFLOW-2497 URL: https://issues.apache.org/jira/browse/AIRFLOW-2497 Project: Apache Airflow Issue Type: Bug Reporter: Kevin Yang >From >[https://github.com/apache/incubator-airflow/blob/master/airflow/task/task_runner/base_task_runner.py#L79-L84,] > only PYTHONPATH is propagated to the child process, which make the behavior >of bash task runner and cgroup task runner different as bash task runner would >issue a `bash -c` command that automatically pass all the env var from the >parent process to the subprocess. Cgroup task runner should not behave >different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
