[
https://issues.apache.org/jira/browse/AIRFLOW-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17233154#comment-17233154
]
ASF GitHub Bot commented on AIRFLOW-5115:
-----------------------------------------
dsynkov opened a new pull request #12389:
URL: https://github.com/apache/airflow/pull/12389
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Fixes a bug in the `S3KeySensor` that prevents the use of Jinja-templated
strings as arguments to `bucket_key` and `bucket_name` by performing a URL
parsing validation in the constructor (_before_ the template gets rendered).
This PR simply moves this validation downstream to the `poke` method. This
unblocks users from using Jinja-templated fields in args to the operator.
To the best of my knowledge there is no corresponding issue in Github but I
have linked the Jira issue.
closes: https://issues.apache.org/jira/browse/AIRFLOW-5115
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja
> variables
> ---------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
> Affects Versions: 1.9.0
> Reporter: Dmitriy Synkov
> Assignee: Dmitriy Synkov
> Priority: Minor
> Labels: easyfix, patch
> Fix For: 2.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> In all Airflow operators (which inherit form {{BaseOperator}}) there is a
> {{template_fields}} attribute defined as ["which fields will get
> jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]).
> For the {{S3KeySensor}} op in specific, these are {{template_fields =
> ('bucket_key', 'bucket_name')}}.
> The {{bucket_key}} kwarg, however, has some input validation in that the
> {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this exception
> is thrown by the
> [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74],
> which makes it impossible to use Jinja strings as an arg to {{bucket_key}},
> since these don't get rendered in the scope of the DAG {{*.py}} file itself.
> Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:
> Given the below DAG code, where "my_s3_key" is
> {{s3://bucket/prefix/object.txt:}}
> {code:java}
> dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))
> s3_variable_sensor = S3KeySensor(
> task_id='s3_variable_sensor',
> bucket_key=Variable.get('my_s3_key'),
> dag=dag
> )
> s3_jinja_sensor = S3KeySensor(
> task_id='s3_jinja_sensor',
> bucket_key="{{ var.value.my_s3_key }}",
> dag=dag
> )
> {code}
> Executing the first task will run just fine while the next task will throw
> the following exception:
> {code:java}
> airflow.exceptions.AirflowException: Please provide a bucket_name.
> {code}
> This ticket is to propose a code change that will move input validation out
> of the constructor to allow for Jinja-templated strings to be passed into
> both {{bucket_name}} and {{bucket_key}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
