[GitHub] [airflow] mik-laj opened a new pull request #7480: [AIRFLOW-6861] Remove pool full on TaskInstance
mik-laj opened a new pull request #7480: [AIRFLOW-6861] Remove pool full on TaskInstance URL: https://github.com/apache/airflow/pull/7480 It's a dead code. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6861) Remove pool_full on TaskInstance
Kamil Bregula created AIRFLOW-6861: -- Summary: Remove pool_full on TaskInstance Key: AIRFLOW-6861 URL: https://issues.apache.org/jira/browse/AIRFLOW-6861 Project: Apache Airflow Issue Type: Bug Components: core Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on a change in pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor
potiuk commented on a change in pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor URL: https://github.com/apache/airflow/pull/7479#discussion_r382440843 ## File path: tests/jobs/test_scheduler_job.py ## @@ -2445,8 +2443,7 @@ def test_retry_still_in_executor(self): but is still present in the executor. """ executor = MockExecutor(do_update=False) -dagbag = DagBag(executor=executor, dag_folder=os.path.join(settings.DAGS_FOLDER, - "no_dags.py")) +dagbag = DagBag(dag_folder=os.path.join(settings.DAGS_FOLDER, "no_dags.py")) dagbag.dags.clear() dagbag.executor = executor Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor
potiuk commented on a change in pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor URL: https://github.com/apache/airflow/pull/7479#discussion_r382440112 ## File path: tests/jobs/test_scheduler_job.py ## @@ -2390,8 +2389,7 @@ def test_scheduler_reschedule(self): """ executor = MockExecutor(do_update=False) -dagbag = DagBag(executor=executor, dag_folder=os.path.join(settings.DAGS_FOLDER, - "no_dags.py")) +dagbag = DagBag(dag_folder=os.path.join(settings.DAGS_FOLDER, "no_dags.py")) dagbag.dags.clear() dagbag.executor = executor Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-589539027 I actually am going to try a slightly different, nicer approach with better separation between the main and backport packages. Bear with me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk edited a comment on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-589539027 I actually am going to try a slightly different, nicer approach with better separation between the main and backport packages. Bare with me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default
potiuk commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default URL: https://github.com/apache/airflow/pull/7450#issuecomment-589538133 I think we will eventually have to slightly modify the source code of the operators when preparing backport packages indeed and In this case I agree monkey patching is worse idea :). The idea is to have class deriving from the BaseOperator and implement MetaClass and have all the backported operators to derive from this class instead. We can do it by modifying the source code (slightly hacky) or by modifying the Concrete Syntax Tree via https://pybowler.io/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor
mik-laj opened a new pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor URL: https://github.com/apache/airflow/pull/7479 It's a dead code. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6858) Decouple DagBag and Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041598#comment-17041598 ] ASF GitHub Bot commented on AIRFLOW-6858: - mik-laj commented on pull request #7479: [AIRFLOW-6858] Decouple DagBag and Executor URL: https://github.com/apache/airflow/pull/7479 It's a dead code. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Decouple DagBag and Executor > > > Key: AIRFLOW-6858 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6858 > Project: Apache Airflow > Issue Type: Bug > Components: core, DAG, executor-kubernetes >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Daniel Imberman >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6860) Default ignore_first_depends_on_past to True
Ping Zhang created AIRFLOW-6860: --- Summary: Default ignore_first_depends_on_past to True Key: AIRFLOW-6860 URL: https://issues.apache.org/jira/browse/AIRFLOW-6860 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 1.10.9 Reporter: Ping Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] commented on issue #6499: [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5704]
stale[bot] commented on issue #6499: [AIRFLOW-5842] Switch to Debian buster image. Depends on [AIRFLOW-5704] URL: https://github.com/apache/airflow/pull/6499#issuecomment-589505057 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4453) none_failed trigger rule cascading skipped state to downstream tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041484#comment-17041484 ] ASF GitHub Bot commented on AIRFLOW-4453: - TV4Fun commented on pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > none_failed trigger rule cascading skipped state to downstream tasks > > > Key: AIRFLOW-4453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4453 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, scheduler >Affects Versions: 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7 >Reporter: Dmytro Kulyk >Assignee: Kaxil Naik >Priority: Major > Labels: skipped > Fix For: 1.10.5 > > Attachments: 3_step.png, cube_update.py, > image-2019-05-02-18-11-28-307.png, simple_skip.png > > > Task with trigger_rule = 'none_failed' cascading *skipped *status to > downstream task > * task have multiple upstream tasks > * trigger_rule set to 'none_failed' > * some of upstream tasks can be skipped due to *latest only* > Basing on documentation this shouldn't happen > !image-2019-05-02-18-11-28-307.png|width=655,height=372! > DAG attached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation
TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098#issuecomment-589475906 Closing this in favor of #7464. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TV4Fun closed pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation
TV4Fun closed pull request #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7477: [AIRFLOW-6857][depends on AIRFLOW-6856] Bulk sync DAGs
codecov-io commented on issue #7477: [AIRFLOW-6857][depends on AIRFLOW-6856] Bulk sync DAGs URL: https://github.com/apache/airflow/pull/7477#issuecomment-589466612 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=h1) Report > Merging [#7477](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `98%`. [](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #7477 +/- ## = - Coverage 86.68% 86.4% -0.28% = Files 882 882 Lines 41526 41608 +82 = - Hits35997 35952 -45 - Misses 55295656 +127 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.22% <ø> (+0.1%)` | :arrow_up: | | [airflow/dag/base\_dag.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9kYWcvYmFzZV9kYWcucHk=) | `69.56% <ø> (+1.56%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.68% <100%> (+0.01%)` | :arrow_up: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.29% <100%> (-0.02%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.06% <97.56%> (+0.16%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/7477/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=footer). Last update [1a9a9f7...12d6ebe](https://codecov.io/gh/apache/airflow/pull/7477?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6859) Airflow scheduler looking for logs of example DAGs in the wrong folder
Abhilash Kishore created AIRFLOW-6859:
-
Summary: Airflow scheduler looking for logs of example DAGs in the
wrong folder
Key: AIRFLOW-6859
URL: https://issues.apache.org/jira/browse/AIRFLOW-6859
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10.9
Reporter: Abhilash Kishore
My AIRFLOW_HOME is set to */home/airflow*
I am running Airflow as a service
`cat /etc/systemd/system/airflow-scheduler.service` :
{code:java}
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service
rabbitmq-server.service mssql-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
mssql-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/local/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
{code}
*BUG:*
Airflow scheduler looking for logs of example DAGs in the wrong folder. Example:
{code:java}
Feb 21 01:57:24 airflow-poc airflow[25847]: Process DagFileProcessor281-Process:
Feb 21 01:57:24 airflow-poc airflow[25847]: Traceback (most recent call last):
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Feb 21 01:57:24 airflow-poc airflow[25847]: self.run()
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
Feb 21 01:57:24 airflow-poc airflow[25847]: self._target(*self._args,
**self._kwargs)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line
135, in _run_file_processor
Feb 21 01:57:24 airflow-poc airflow[25847]: set_context(log, file_path)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/utils/log/logging_mixin.py",
line 198, in set_context
Feb 21 01:57:24 airflow-poc airflow[25847]: handler.set_context(value)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/utils/log/file_processor_handler.py",
line 66, in set_context
Feb 21 01:57:24 airflow-poc airflow[25847]: self.handler =
logging.FileHandler(local_loc)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__
Feb 21 01:57:24 airflow-poc airflow[25847]: StreamHandler.__init__(self,
self._open())
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
Feb 21 01:57:24 airflow-poc airflow[25847]: return open(self.baseFilename,
self.mode, encoding=self.encoding)
Feb 21 01:57:24 airflow-poc airflow[25847]: PermissionError: [Errno 13]
Permission denied:
'/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log'{code}
The same is happening for all example DAGs (haven't loaded any other DAGs yet).
Airflow scheduler is looking for the logs of the example DAG in
`/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`.
It should instead look in
`/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`
instead (without the '/home/airflow' prefix).
In the `dag` table in the database, the DAG with dag_id `example_skip_dag.` has
fileloc
`/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py`.
Not sure why Airflow scheduler is appending `/home/airflow' to this path
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj closed pull request #7478: Remove pool full
mik-laj closed pull request #7478: Remove pool full URL: https://github.com/apache/airflow/pull/7478 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #7478: Remove pool full
mik-laj opened a new pull request #7478: Remove pool full URL: https://github.com/apache/airflow/pull/7478 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default
mik-laj commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default URL: https://github.com/apache/airflow/pull/7450#issuecomment-589449506 We can do it without monkeypatching. All we have to do is deliver a new BaseOperator class together with operators. We can do it by slightly modifying the source code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6858) Decouple DagBag and Executor
Kamil Bregula created AIRFLOW-6858: -- Summary: Decouple DagBag and Executor Key: AIRFLOW-6858 URL: https://issues.apache.org/jira/browse/AIRFLOW-6858 Project: Apache Airflow Issue Type: Bug Components: core, DAG, executor-kubernetes Affects Versions: 1.10.9 Reporter: Kamil Bregula Assignee: Daniel Imberman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6821) When task is marked as success through UI, success callback is not triggered
[ https://issues.apache.org/jira/browse/AIRFLOW-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041376#comment-17041376 ] ASF subversion and git services commented on AIRFLOW-6821: -- Commit 6cd37da0fd3f5a6966bd321db63123da4abc5a54 in airflow's branch refs/heads/master from Saurabh [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6cd37da ] [AIRFLOW-6821] Success callback not called when task marked as success from UI (#7447) > When task is marked as success through UI, success callback is not triggered > > > Key: AIRFLOW-6821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6821 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Saurabh Dhupar >Assignee: Saurabh Dhupar >Priority: Minor > Fix For: 2.0.0, 1.10.10 > > > When you mark a task as success - Once Worker detects and stops the process > but - on_success_callback is not called. > In this case, I would expect on_success_callbackto be called to complete it's > intended functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7447: [AIRFLOW-6821] - Success callback not called when task marked as succ…
kaxil merged pull request #7447: [AIRFLOW-6821] - Success callback not called when task marked as succ… URL: https://github.com/apache/airflow/pull/7447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7447: [AIRFLOW-6821] - Success callback not called when task marked as succ…
boring-cyborg[bot] commented on issue #7447: [AIRFLOW-6821] - Success callback not called when task marked as succ… URL: https://github.com/apache/airflow/pull/7447#issuecomment-589428802 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6857) Bulk sync DAGs
[
https://issues.apache.org/jira/browse/AIRFLOW-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041382#comment-17041382
]
ASF GitHub Bot commented on AIRFLOW-6857:
-
mik-laj commented on pull request #7477: [AIRFLOW-6857][depends on
AIRFLOW-6856] Bulk sync DAGs
URL: https://github.com/apache/airflow/pull/7477
I created the following DAG file:
```python
args = {
'owner': 'airflow',
'start_date': days_ago(3),
}
def create_dag(dag_number):
dag = DAG(
dag_id=f'perf_50_dag_dummy_tasks_{dag_number}_of_50',
default_args=args,
schedule_interval=None,
dagrun_timeout=timedelta(minutes=60)
)
for j in range(1, 5):
DummyOperator(
task_id='task_{}_of_5'.format(j),
dag=dag
)
return dag
for i in range(1, 200):
globals()[f"dag_{i}"] = create_dag(i)
```
and I used the following code to test performance.
```python
import functools
import logging
import time
from airflow.jobs.scheduler_job import DagFileProcessor
class CountQueries(object):
def __init__(self):
self.count = 0
def __enter__(self):
from sqlalchemy import event
from airflow.settings import engine
event.listen(engine, "after_cursor_execute",
self.after_cursor_execute)
return None
def after_cursor_execute(self, *args, **kwargs):
self.count += 1
def __exit__(self, type, value, traceback):
from sqlalchemy import event
from airflow.settings import engine
event.remove(engine, "after_cursor_execute",
self.after_cursor_execute)
print('Query count: ', self.count)
count_queries = CountQueries
DAG_FILE = "/files/dags/200_dag_5_dummy_tasks.py"
log = logging.getLogger("airflow.processor")
processor = DagFileProcessor([], log)
def timing(f):
@functools.wraps(f)
def wrap(*args):
RETRY_COUNT = 5
r = []
for i in range(RETRY_COUNT):
time1 = time.time()
f(*args)
time2 = time.time()
diff = (time2 - time1) * 1000.0
r.append(diff)
# print('Retry %d took %0.3f ms' % (i, diff))
print('Average took %0.3f ms' % (sum(r) / RETRY_COUNT))
return wrap
@timing
def slow_case():
with count_queries():
processor.process_file(DAG_FILE, None, pickle_dags=False)
slow_case()
```
I also cherry-picked AIRFLOW-6856
As a result, I obtained the following values
**Master**:
Query count: 1792
Average took 4505.891 ms
**AIRFLOW-6856:**
Query count: 1197
Average took 3203.710 ms
**Current:**
Query count: 602
Average time: 2018.891 ms
**Diff to AIRFLOW-6856**
Query count: -1190 (-66%)
Average time: -1185 ms (-36%)
**Diff to master*
Query count: -592 (-49%)
Average time: -2484 ms (-55%)
Thanks for support to @evgenyshulman from Databand!
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] [airflow] mik-laj opened a new pull request #7477: [AIRFLOW-6857][depends on AIRFLOW-6856] Bulk sync DAGs
mik-laj opened a new pull request #7477: [AIRFLOW-6857][depends on
AIRFLOW-6856] Bulk sync DAGs
URL: https://github.com/apache/airflow/pull/7477
I created the following DAG file:
```python
args = {
'owner': 'airflow',
'start_date': days_ago(3),
}
def create_dag(dag_number):
dag = DAG(
dag_id=f'perf_50_dag_dummy_tasks_{dag_number}_of_50',
default_args=args,
schedule_interval=None,
dagrun_timeout=timedelta(minutes=60)
)
for j in range(1, 5):
DummyOperator(
task_id='task_{}_of_5'.format(j),
dag=dag
)
return dag
for i in range(1, 200):
globals()[f"dag_{i}"] = create_dag(i)
```
and I used the following code to test performance.
```python
import functools
import logging
import time
from airflow.jobs.scheduler_job import DagFileProcessor
class CountQueries(object):
def __init__(self):
self.count = 0
def __enter__(self):
from sqlalchemy import event
from airflow.settings import engine
event.listen(engine, "after_cursor_execute",
self.after_cursor_execute)
return None
def after_cursor_execute(self, *args, **kwargs):
self.count += 1
def __exit__(self, type, value, traceback):
from sqlalchemy import event
from airflow.settings import engine
event.remove(engine, "after_cursor_execute",
self.after_cursor_execute)
print('Query count: ', self.count)
count_queries = CountQueries
DAG_FILE = "/files/dags/200_dag_5_dummy_tasks.py"
log = logging.getLogger("airflow.processor")
processor = DagFileProcessor([], log)
def timing(f):
@functools.wraps(f)
def wrap(*args):
RETRY_COUNT = 5
r = []
for i in range(RETRY_COUNT):
time1 = time.time()
f(*args)
time2 = time.time()
diff = (time2 - time1) * 1000.0
r.append(diff)
# print('Retry %d took %0.3f ms' % (i, diff))
print('Average took %0.3f ms' % (sum(r) / RETRY_COUNT))
return wrap
@timing
def slow_case():
with count_queries():
processor.process_file(DAG_FILE, None, pickle_dags=False)
slow_case()
```
I also cherry-picked AIRFLOW-6856
As a result, I obtained the following values
**Master**:
Query count: 1792
Average took 4505.891 ms
**AIRFLOW-6856:**
Query count: 1197
Average took 3203.710 ms
**Current:**
Query count: 602
Average time: 2018.891 ms
**Diff to AIRFLOW-6856**
Query count: -1190 (-66%)
Average time: -1185 ms (-36%)
**Diff to master*
Query count: -592 (-49%)
Average time: -2484 ms (-55%)
Thanks for support to @evgenyshulman from Databand!
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6857) Bulk sync DAGs
Kamil Bregula created AIRFLOW-6857: -- Summary: Bulk sync DAGs Key: AIRFLOW-6857 URL: https://issues.apache.org/jira/browse/AIRFLOW-6857 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6821) When task is marked as success through UI, success callback is not triggered
[ https://issues.apache.org/jira/browse/AIRFLOW-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-6821. - Fix Version/s: 1.10.10 2.0.0 Resolution: Fixed > When task is marked as success through UI, success callback is not triggered > > > Key: AIRFLOW-6821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6821 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Saurabh Dhupar >Assignee: Saurabh Dhupar >Priority: Minor > Fix For: 2.0.0, 1.10.10 > > > When you mark a task as success - Once Worker detects and stops the process > but - on_success_callback is not called. > In this case, I would expect on_success_callbackto be called to complete it's > intended functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6821) When task is marked as success through UI, success callback is not triggered
[ https://issues.apache.org/jira/browse/AIRFLOW-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041375#comment-17041375 ] ASF GitHub Bot commented on AIRFLOW-6821: - kaxil commented on pull request #7447: [AIRFLOW-6821] - Success callback not called when task marked as succ… URL: https://github.com/apache/airflow/pull/7447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > When task is marked as success through UI, success callback is not triggered > > > Key: AIRFLOW-6821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6821 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Saurabh Dhupar >Assignee: Saurabh Dhupar >Priority: Minor > > When you mark a task as success - Once Worker detects and stops the process > but - on_success_callback is not called. > In this case, I would expect on_success_callbackto be called to complete it's > intended functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-589395121 looks like the error linked above was one off - a later build didn't trigger it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7476: [AIRFLOW-6856] Bulk fetch paused_dag_ids
codecov-io edited a comment on issue #7476: [AIRFLOW-6856] Bulk fetch paused_dag_ids URL: https://github.com/apache/airflow/pull/7476#issuecomment-589383726 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=h1) Report > Merging [#7476](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #7476 +/- ## = - Coverage 86.68% 86.4% -0.29% = Files 882 882 Lines 41526 41593 +67 = - Hits35997 35937 -60 - Misses 55295656 +127 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.22% <ø> (+0.1%)` | :arrow_up: | | [airflow/dag/base\_dag.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9kYWcvYmFzZV9kYWcucHk=) | `69.56% <ø> (+1.56%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.69% <100%> (+0.03%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | ... and [15 more](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=footer). Last update [1a9a9f7...5b33dc5](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default
ashb commented on issue #7450: [AIRFLOW-6829][WIP] Introduce BaseOperatorMetaClass - auto-apply apply_default URL: https://github.com/apache/airflow/pull/7450#issuecomment-589384032 It's horrible, but could we maybe monkey-patch in the metaclass to BashOperator in the back port package? (This is probably a terrible idea even if it's technically possible) We could do it via a plug-in `load` feature already built in to Airflow (it just gives us any automatically called function> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7476: [AIRFLOW-6856] Bulk fetch paused_dag_ids
codecov-io commented on issue #7476: [AIRFLOW-6856] Bulk fetch paused_dag_ids URL: https://github.com/apache/airflow/pull/7476#issuecomment-589383726 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=h1) Report > Merging [#7476](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `0.64%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7476 +/- ## == - Coverage 86.68% 86.04% -0.65% == Files 882 882 Lines 4152641593 +67 == - Hits3599735788 -209 - Misses 5529 5805 +276 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `85.71% <ø> (-2.41%)` | :arrow_down: | | [airflow/dag/base\_dag.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9kYWcvYmFzZV9kYWcucHk=) | `69.56% <ø> (+1.56%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.11% <100%> (-0.56%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0%> (-60.72%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | ... and [15 more](https://codecov.io/gh/apache/airflow/pull/7476/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=footer). Last update [1a9a9f7...5b33dc5](https://codecov.io/gh/apache/airflow/pull/7476?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-6840. Fix Version/s: 1.10.10 Resolution: Fixed > Bump up version of future to 0.18.2 in v-10-testing to get closer to python > 3.8 > --- > > Key: AIRFLOW-6840 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6840 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > Bump up future from 0.17 to 0.18.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb merged pull request #7471: [AIRFLOW-6840] Bump up version of future
ashb merged pull request #7471: [AIRFLOW-6840] Bump up version of future URL: https://github.com/apache/airflow/pull/7471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041331#comment-17041331 ] ASF GitHub Bot commented on AIRFLOW-6840: - ashb commented on pull request #7471: [AIRFLOW-6840] Bump up version of future URL: https://github.com/apache/airflow/pull/7471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump up version of future to 0.18.2 in v-10-testing to get closer to python > 3.8 > --- > > Key: AIRFLOW-6840 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6840 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > > Bump up future from 0.17 to 0.18.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041332#comment-17041332 ] ASF subversion and git services commented on AIRFLOW-6840: -- Commit b79bbb05752a291c5d8352c8bbbd14120e90e69d in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b79bbb0 ] [AIRFLOW-6840] Bump up version of future (#7471) > Bump up version of future to 0.18.2 in v-10-testing to get closer to python > 3.8 > --- > > Key: AIRFLOW-6840 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6840 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.9 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > > Bump up future from 0.17 to 0.18.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor
ashb commented on a change in pull request #7163: [AIRFLOW-6542] add
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r382290570
##
File path:
airflow/providers/cncf/kubernetes/operators/spark_kubernetes_operator.py
##
@@ -0,0 +1,83 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+import yaml
+from kubernetes import client
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes_hook import
Kuberneteshook
+from airflow.utils.decorators import apply_defaults
+
+
+class SparkKubernetesOperator(BaseOperator):
+"""
+Creates sparkApplication object in kubernetes cluster:
+ .. seealso::
+For more detail about Spark Application Object have a look at the
reference:
+
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/api-docs.md#sparkapplication
+
+:param sparkapplication_file: filepath to kubernetes
custom_resource_definition of sparkApplication
+:type sparkapplication_file: str
+:param namespace: kubernetes namespace to put sparkApplication
+:type namespace: str
+:param conn_id: the connection to Kubernetes cluster
+:type conn_id: str
+"""
+
+template_fields = ['sparkapplication_file', 'namespace']
+template_ext = ('yaml', 'yml', 'json')
+ui_color = '#f4a460'
+
+@apply_defaults
+def __init__(self,
+ sparkapplication_file: str,
+ namespace: Optional[str] = None,
+ conn_id: str = 'kubernetes_default',
+ *args, **kwargs) -> None:
+super().__init__(*args, **kwargs)
+self.sparkapplication_file = sparkapplication_file
+self.namespace = namespace
+self.conn_id = conn_id
+
+def execute(self, context):
+self.log.info("Creating sparkApplication")
+hook = Kuberneteshook(conn_id=self.conn_id)
+api_client = hook.get_conn()
+api = client.CustomObjectsApi(api_client)
Review comment:
Oh sorry I was misreading this code - `client` was confusing me as a module
name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor
ashb commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor URL: https://github.com/apache/airflow/pull/7163#discussion_r382289306 ## File path: airflow/providers/cncf/kubernetes/hooks/kubernetes_hook.py ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import tempfile + +from kubernetes import client, config + +from airflow.hooks.base_hook import BaseHook + + +class Kuberneteshook(BaseHook): Review comment: Non, this PR can stay as it is, just talking about future work. KubePodOp already works against external clusters though This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor
ashb commented on a change in pull request #7163: [AIRFLOW-6542] add
spark-on-k8s operator/hook/sensor
URL: https://github.com/apache/airflow/pull/7163#discussion_r382288655
##
File path: airflow/providers/cncf/kubernetes/hooks/kubernetes_hook.py
##
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tempfile
+
+from kubernetes import client, config
+
+from airflow.hooks.base_hook import BaseHook
+
+
+class Kuberneteshook(BaseHook):
+"""
+Creates Kubernetes API connection.
+
+:param conn_id: the connection to Kubernetes cluster
+"""
+
+def __init__(
+self,
+conn_id="kubernetes_default"
+):
+self.connection = self.get_connection(conn_id)
+self.extras = self.connection.extra_dejson
+
+def get_conn(self):
+"""
+Returns kubernetes api session for use with requests
+"""
+
+if self._get_field(("in_cluster")):
+self.log.debug("loading kube_config from: in_cluster
configuration")
+config.load_incluster_config()
+elif self._get_field("kube_config") is None or
self._get_field("kube_config") == '':
+self.log.debug("loading kube_config from: default file")
+else:
+with tempfile.NamedTemporaryFile() as temp_config:
+self.log.debug("loading kube_config from: connection
kube_config")
+temp_config.write(self._get_field("kube_config").encode())
+config.load_kube_config(temp_config.name)
+temp_config.flush()
Review comment:
There is zero point flushing after kube has read from it - that does nothing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6717: [AIRFLOW-6160] Move _check_and_change_state_before_execution to LocalTaskJob
stale[bot] commented on issue #6717: [AIRFLOW-6160] Move _check_and_change_state_before_execution to LocalTaskJob URL: https://github.com/apache/airflow/pull/6717#issuecomment-589378752 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests URL: https://github.com/apache/airflow/pull/7389#issuecomment-589365981 also applied the comments from @efedotova ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests URL: https://github.com/apache/airflow/pull/7389#issuecomment-589362536 rebased and hopefully all good now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6856) Bulk fetch paused_dag_ids
[
https://issues.apache.org/jira/browse/AIRFLOW-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041311#comment-17041311
]
ASF GitHub Bot commented on AIRFLOW-6856:
-
mik-laj commented on pull request #7476: [AIRFLOW-6856] Bulk fetch
paused_dag_ids
URL: https://github.com/apache/airflow/pull/7476
I created the following DAG file:
```python
args = {
'owner': 'airflow',
'start_date': days_ago(3),
}
def create_dag(dag_number):
dag = DAG(
dag_id=f'perf_50_dag_dummy_tasks_{dag_number}_of_50',
default_args=args,
schedule_interval=None,
dagrun_timeout=timedelta(minutes=60)
)
for j in range(1, 5):
DummyOperator(
task_id='task_{}_of_5'.format(j),
dag=dag
)
return dag
for i in range(1, 50):
globals()[f"dag_{i}"] = create_dag(i)
```
and I used the following code to test performance.
```python
import functools
import logging
import time
from airflow.jobs.scheduler_job import DagFileProcessor
class CountQueries(object):
def __init__(self):
self.count = 0
def __enter__(self):
from sqlalchemy import event
from airflow.settings import engine
event.listen(engine, "after_cursor_execute",
self.after_cursor_execute)
return None
def after_cursor_execute(self, *args, **kwargs):
self.count += 1
def __exit__(self, type, value, traceback):
from sqlalchemy import event
from airflow.settings import engine
event.remove(engine, "after_cursor_execute",
self.after_cursor_execute)
print('Query count: ', self.count)
count_queries = CountQueries
DAG_FILE = "/files/dags/50_dag_5_dummy_tasks.py"
log = logging.getLogger("airflow.processor")
processor = DagFileProcessor([], log)
def timing(f):
@functools.wraps(f)
def wrap(*args):
RETRY_COUNT = 5
r = []
for i in range(RETRY_COUNT):
time1 = time.time()
f(*args)
time2 = time.time()
diff = (time2 - time1) * 1000.0
r.append(diff)
# print('Retry %d took %0.3f ms' % (i, diff))
print('Average took %0.3f ms' % (sum(r) / RETRY_COUNT))
return wrap
@timing
def slow_case():
with count_queries():
processor.process_file(DAG_FILE, None, pickle_dags=False)
slow_case()
```
As a result, I obtained the following results
**Before:**
Query count: 442
Average time: 1182.187 ms
**After:**
Query count: 297
Average time: 769.421 ms
**Diff:**
Query count: -145 (-32%)
Average time: -413 ms (-34%)
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Bulk fetch paused_dag_ids
> -
>
> Key: AIRFLOW-6856
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6856
> Project: Apache Airflow
> Issue Type
[GitHub] [airflow] mik-laj opened a new pull request #7476: [AIRFLOW-6856] Bulk fetch paused_dag_ids
mik-laj opened a new pull request #7476: [AIRFLOW-6856] Bulk fetch
paused_dag_ids
URL: https://github.com/apache/airflow/pull/7476
I created the following DAG file:
```python
args = {
'owner': 'airflow',
'start_date': days_ago(3),
}
def create_dag(dag_number):
dag = DAG(
dag_id=f'perf_50_dag_dummy_tasks_{dag_number}_of_50',
default_args=args,
schedule_interval=None,
dagrun_timeout=timedelta(minutes=60)
)
for j in range(1, 5):
DummyOperator(
task_id='task_{}_of_5'.format(j),
dag=dag
)
return dag
for i in range(1, 50):
globals()[f"dag_{i}"] = create_dag(i)
```
and I used the following code to test performance.
```python
import functools
import logging
import time
from airflow.jobs.scheduler_job import DagFileProcessor
class CountQueries(object):
def __init__(self):
self.count = 0
def __enter__(self):
from sqlalchemy import event
from airflow.settings import engine
event.listen(engine, "after_cursor_execute",
self.after_cursor_execute)
return None
def after_cursor_execute(self, *args, **kwargs):
self.count += 1
def __exit__(self, type, value, traceback):
from sqlalchemy import event
from airflow.settings import engine
event.remove(engine, "after_cursor_execute",
self.after_cursor_execute)
print('Query count: ', self.count)
count_queries = CountQueries
DAG_FILE = "/files/dags/50_dag_5_dummy_tasks.py"
log = logging.getLogger("airflow.processor")
processor = DagFileProcessor([], log)
def timing(f):
@functools.wraps(f)
def wrap(*args):
RETRY_COUNT = 5
r = []
for i in range(RETRY_COUNT):
time1 = time.time()
f(*args)
time2 = time.time()
diff = (time2 - time1) * 1000.0
r.append(diff)
# print('Retry %d took %0.3f ms' % (i, diff))
print('Average took %0.3f ms' % (sum(r) / RETRY_COUNT))
return wrap
@timing
def slow_case():
with count_queries():
processor.process_file(DAG_FILE, None, pickle_dags=False)
slow_case()
```
As a result, I obtained the following results
**Before:**
Query count: 442
Average time: 1182.187 ms
**After:**
Query count: 297
Average time: 769.421 ms
**Diff:**
Query count: -145 (-32%)
Average time: -413 ms (-34%)
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6856) Bulk fetch paused_dag_ids
Kamil Bregula created AIRFLOW-6856: -- Summary: Bulk fetch paused_dag_ids Key: AIRFLOW-6856 URL: https://issues.apache.org/jira/browse/AIRFLOW-6856 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
ashb commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-589294087 Ahha. Nothing about python packaging not following docs surprises me anymore This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages
potiuk commented on issue #7391: [AIRFLOW-6663] Prepare backporting packages URL: https://github.com/apache/airflow/pull/7391#issuecomment-589293493 All fixed I think @ashb -> the warning about license files was also in the original bdist_wheel. It seems that (against documentation) bdist_wheel cannot handle * glob in the name of the license files (from setup.cfg). I fixed it by generating the list of license file in pre-commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq …
codecov-io edited a comment on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq … URL: https://github.com/apache/airflow/pull/7475#issuecomment-589281813 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=h1) Report > Merging [#7475](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `0.62%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7475 +/- ## == - Coverage 86.68% 86.06% -0.63% == Files 882 882 Lines 4152641600 +74 == - Hits3599735801 -196 - Misses 5529 5799 +270 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...roviders/google/cloud/operators/gcs\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3NfdG9fYmlncXVlcnkucHk=) | `67.6% <0%> (-2.99%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0%> (-38.47%)` | :arrow_down: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0%> (-37.84%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=footer). Last update [1a9a9f7...d0c42bb](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-589285097 Quick question - I don't see any reference to the singularity operator [here](https://travis-ci.org/apache/airflow/jobs/653112190) but the tests failed - do you know what's up? I'll focus on just the vanilla postgres for now, that seems to have errors for the operator. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq …
codecov-io edited a comment on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq … URL: https://github.com/apache/airflow/pull/7475#issuecomment-589281813 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=h1) Report > Merging [#7475](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `1.08%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7475 +/- ## == - Coverage 86.68% 85.59% -1.09% == Files 882 882 Lines 4152641600 +74 == - Hits3599735608 -389 - Misses 5529 5992 +463 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...roviders/google/cloud/operators/gcs\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3NfdG9fYmlncXVlcnkucHk=) | `67.6% <0%> (-2.99%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0%> (-60.72%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=footer). Last update [1a9a9f7...d0c42bb](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq …
codecov-io commented on issue #7475: [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq … URL: https://github.com/apache/airflow/pull/7475#issuecomment-589281813 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=h1) Report > Merging [#7475](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1a9a9f7618f1c22e3e9a6ef4ec73b717c7760c7d?src=pr&el=desc) will **decrease** coverage by `1.08%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7475 +/- ## == - Coverage 86.68% 85.59% -1.09% == Files 882 882 Lines 4152641600 +74 == - Hits3599735608 -389 - Misses 5529 5992 +463 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...roviders/google/cloud/operators/gcs\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9nY3NfdG9fYmlncXVlcnkucHk=) | `67.6% <0%> (-2.99%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0%> (-60.72%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/7475/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=footer). Last update [1a9a9f7...d0c42bb](https://codecov.io/gh/apache/airflow/pull/7475?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mtagle opened a new pull request #7475: AIRFLOW-6855: Escape project_dataset_table in SQL query in gcs to bq …
mtagle opened a new pull request #7475: AIRFLOW-6855: Escape project_dataset_table in SQL query in gcs to bq … URL: https://github.com/apache/airflow/pull/7475 …operator Without escaping, if the project is specified in project_dataset_table and contains a -, the query will fail with an error. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6855) GCS to BQ load will fail with syntax error if specified project has a - in it
[
https://issues.apache.org/jira/browse/AIRFLOW-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041254#comment-17041254
]
ASF GitHub Bot commented on AIRFLOW-6855:
-
mtagle commented on pull request #7475: AIRFLOW-6855: Escape
project_dataset_table in SQL query in gcs to bq …
URL: https://github.com/apache/airflow/pull/7475
…operator
Without escaping, if the project is specified in project_dataset_table and
contains a -,
the query will fail with an error.
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> GCS to BQ load will fail with syntax error if specified project has a - in it
> -
>
> Key: AIRFLOW-6855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6855
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.9
>Reporter: Moira Tagle
>Assignee: Moira Tagle
>Priority: Minor
>
> If you try to run a GCS -> BQ load that explicitly specifies a destination
> project that contains a -, the sql that runs to emit the final log message:
> SELECT MAX(\{id}) FROM \{project_dataset_table}
> will fail because the - in the project is not escaped in the SQL.
> datasets and tables cannot have - in them, and if you are implicitly getting
> the project from the configured connection it will not explicitly be in the
> SQL, so this is only a problem if the project is explicitly specified and it
> has a - in it.
> To fix this, the project_dataset_table should be escaped in the SQL query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6855) GCS to BQ load will fail with syntax error if specified project has a - in it
Moira Tagle created AIRFLOW-6855:
Summary: GCS to BQ load will fail with syntax error if specified
project has a - in it
Key: AIRFLOW-6855
URL: https://issues.apache.org/jira/browse/AIRFLOW-6855
Project: Apache Airflow
Issue Type: Bug
Components: gcp
Affects Versions: 1.10.9
Reporter: Moira Tagle
Assignee: Moira Tagle
If you try to run a GCS -> BQ load that explicitly specifies a destination
project that contains a -, the sql that runs to emit the final log message:
SELECT MAX(\{id}) FROM \{project_dataset_table}
will fail because the - in the project is not escaped in the SQL.
datasets and tables cannot have - in them, and if you are implicitly getting
the project from the configured connection it will not explicitly be in the
SQL, so this is only a problem if the project is explicitly specified and it
has a - in it.
To fix this, the project_dataset_table should be escaped in the SQL query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages
codecov-io commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages URL: https://github.com/apache/airflow/pull/7456#issuecomment-589224098 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=h1) Report > Merging [#7456](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/aff3a361b4092212c0757f9ce88fa2e40d25d1f4?src=pr&el=desc) will **decrease** coverage by `78.78%`. > The diff coverage is `5.55%`. [](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master #7456 +/- ## == - Coverage 86.57% 7.79% -78.79% == Files 882 882 Lines 41597 41608 +11 == - Hits360133243-32770 - Misses 5584 38365+32781 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/providers/imap/hooks/imap.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvaW1hcC9ob29rcy9pbWFwLnB5) | `0% <0%> (-94.92%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `40.36% <0%> (-56.15%)` | :arrow_down: | | [airflow/providers/dingding/hooks/dingding.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZGluZ2RpbmcvaG9va3MvZGluZ2RpbmcucHk=) | `0% <0%> (-66.67%)` | :arrow_down: | | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [.../providers/google/cloud/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `0% <0%> (-95.9%)` | :arrow_down: | | [airflow/providers/opsgenie/hooks/opsgenie\_alert.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvb3BzZ2VuaWUvaG9va3Mvb3BzZ2VuaWVfYWxlcnQucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [.../cloud/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zdG9yYWdlX3RyYW5zZmVyX3NlcnZpY2UucHk=) | `0% <0%> (-95.64%)` | :arrow_down: | | [...ow/providers/google/cloud/operators/sftp\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9zZnRwX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/google/cloud/operators/functions.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9mdW5jdGlvbnMucHk=) | `0% <0%> (-98.02%)` | :arrow_down: | | ... and [886 more](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=footer). Last update [aff3a36...18c9202](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8
potiuk commented on issue #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 URL: https://github.com/apache/airflow/pull/7474#issuecomment-589221949 Some static checks to check. Also I think it's a good time to think about pythn 3.8 compatibility This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #7410: [AIRFLOW-6790] Add basic Tableau Integration
feluelle commented on issue #7410: [AIRFLOW-6790] Add basic Tableau Integration URL: https://github.com/apache/airflow/pull/7410#issuecomment-589211314 Ready for another round @Fokko @eladkal. I am going to continue tomorrow :) Thank you for your time! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6851) Logging level: inconsistency between doc/implementation/tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula updated AIRFLOW-6851:
---
Attachment: Screenshot 2020-02-20 at 18.03.45.png
> Logging level: inconsistency between doc/implementation/tests
> -
>
> Key: AIRFLOW-6851
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6851
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
> Attachments: Screenshot 2020-02-20 at 18.03.45.png
>
>
> The documentation at
> [https://airflow.apache.org/docs/stable/configurations-ref.html] only states
> that there is a configuration for the logging level for Core (corresponding
> to AIRFLOW__CORE__LOGGING_LEVEL).
>
> But in the tests (tests/test_configuration.py) and the default configuration
> file another logging level option exists under Logging :
>
> @conf_vars({
> ('logging', 'logging_level'): None,
> ('core', 'logging_level'): None,
> })
>
> And when following the documentation and setting AIRFLOW__CORE__LOGGING_LEVEL
> to WARN there are INFO lines in the logs.
>
> Those inconsistencies should be clarified in the doc, and fixed in the code
> if need be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6851) Logging level: inconsistency between doc/implementation/tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041142#comment-17041142
]
Kamil Bregula commented on AIRFLOW-6851:
!Screenshot 2020-02-20 at 18.03.45.png!
> Logging level: inconsistency between doc/implementation/tests
> -
>
> Key: AIRFLOW-6851
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6851
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
> Attachments: Screenshot 2020-02-20 at 18.03.45.png
>
>
> The documentation at
> [https://airflow.apache.org/docs/stable/configurations-ref.html] only states
> that there is a configuration for the logging level for Core (corresponding
> to AIRFLOW__CORE__LOGGING_LEVEL).
>
> But in the tests (tests/test_configuration.py) and the default configuration
> file another logging level option exists under Logging :
>
> @conf_vars({
> ('logging', 'logging_level'): None,
> ('core', 'logging_level'): None,
> })
>
> And when following the documentation and setting AIRFLOW__CORE__LOGGING_LEVEL
> to WARN there are INFO lines in the logs.
>
> Those inconsistencies should be clarified in the doc, and fixed in the code
> if need be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6851) Logging level: inconsistency between doc/implementation/tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041141#comment-17041141
]
Kamil Bregula commented on AIRFLOW-6851:
We only keep current information in the documentation. Information about the
feature being deprecated is available in UPDATIND.MD. We also display
deprecation warning:
{code:java}
/opt/airflow/airflow/config_templates/airflow_local_settings.py:31:
DeprecationWarning: The logging_level option in [core] has been moved to the
logging_level option in [logging] - the old setting has been used, but please
update your config.{code}
More information:
[https://github.com/apache/airflow/blob/master/UPDATING.md#logging-configuration-has-been-moved-to-new-section]
> Logging level: inconsistency between doc/implementation/tests
> -
>
> Key: AIRFLOW-6851
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6851
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> The documentation at
> [https://airflow.apache.org/docs/stable/configurations-ref.html] only states
> that there is a configuration for the logging level for Core (corresponding
> to AIRFLOW__CORE__LOGGING_LEVEL).
>
> But in the tests (tests/test_configuration.py) and the default configuration
> file another logging level option exists under Logging :
>
> @conf_vars({
> ('logging', 'logging_level'): None,
> ('core', 'logging_level'): None,
> })
>
> And when following the documentation and setting AIRFLOW__CORE__LOGGING_LEVEL
> to WARN there are INFO lines in the logs.
>
> Those inconsistencies should be clarified in the doc, and fixed in the code
> if need be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6851) Logging level: inconsistency between doc/implementation/tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041132#comment-17041132
]
Noël BARDELOT edited comment on AIRFLOW-6851 at 2/20/20 4:55 PM:
-
After analyzing the configuration.py behaviour I understand that the new
[Logging] section's logging_level is deprecating the [Core] similar options.
This is undocumented (those options do not appear as “deprecated” in the
configuration reference).
Plus, the behaviour of logging_level is not consistent with the gunicorn logs
that are INFO whatever airflow's logging level is. This should be documented as
well, and probably clearly stated in the logging_level option's comment in the
config file in my opinion.
was (Author: nbardelot):
After analyzing the configuration.py behaviour I understand that the [Logging]
section's logging_level is deprecated (as are several other options). This is
undocumented (those options do not appear as “deprecated” in the configuration
reference, though they still work).
Plus, the behaviour of [Core]'s logging_level is not consistent with the
gunicorn logs that are INFO whatever airflow's logging level is. This should be
documented as well, and probably clearly stated in the [Core] logging_level
option comment in the config file in my opinion.
> Logging level: inconsistency between doc/implementation/tests
> -
>
> Key: AIRFLOW-6851
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6851
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> The documentation at
> [https://airflow.apache.org/docs/stable/configurations-ref.html] only states
> that there is a configuration for the logging level for Core (corresponding
> to AIRFLOW__CORE__LOGGING_LEVEL).
>
> But in the tests (tests/test_configuration.py) and the default configuration
> file another logging level option exists under Logging :
>
> @conf_vars({
> ('logging', 'logging_level'): None,
> ('core', 'logging_level'): None,
> })
>
> And when following the documentation and setting AIRFLOW__CORE__LOGGING_LEVEL
> to WARN there are INFO lines in the logs.
>
> Those inconsistencies should be clarified in the doc, and fixed in the code
> if need be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6851) Logging level: inconsistency between doc/implementation/tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041132#comment-17041132
]
Noël BARDELOT commented on AIRFLOW-6851:
After analyzing the configuration.py behaviour I understand that the [Logging]
section's logging_level is deprecated (as are several other options). This is
undocumented (those options do not appear as “deprecated” in the configuration
reference, though they still work).
Plus, the behaviour of [Core]'s logging_level is not consistent with the
gunicorn logs that are INFO whatever airflow's logging level is. This should be
documented as well, and probably clearly stated in the [Core] logging_level
option comment in the config file in my opinion.
> Logging level: inconsistency between doc/implementation/tests
> -
>
> Key: AIRFLOW-6851
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6851
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> The documentation at
> [https://airflow.apache.org/docs/stable/configurations-ref.html] only states
> that there is a configuration for the logging level for Core (corresponding
> to AIRFLOW__CORE__LOGGING_LEVEL).
>
> But in the tests (tests/test_configuration.py) and the default configuration
> file another logging level option exists under Logging :
>
> @conf_vars({
> ('logging', 'logging_level'): None,
> ('core', 'logging_level'): None,
> })
>
> And when following the documentation and setting AIRFLOW__CORE__LOGGING_LEVEL
> to WARN there are INFO lines in the logs.
>
> Those inconsistencies should be clarified in the doc, and fixed in the code
> if need be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk commented on a change in pull request #7466: [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst
potiuk commented on a change in pull request #7466: [AIRFLOW-] Update tests info in CONTRIBUTING.rst URL: https://github.com/apache/airflow/pull/7466#discussion_r382125374 ## File path: CONTRIBUTING.rst ## @@ -326,13 +326,13 @@ Test Infrastructure We support the following types of tests: -* **Unit tests** are Python ``nose`` tests launched with ``run-tests``. +* **Unit tests** are Python tests launched with ``pytest``. Unit tests are available both in the `Breeze environment `_ and `local virtualenv `_. * **Integration tests** are available in the Breeze development environment that is also used for Airflow Travis CI tests. Integration test are special tests that require - additional services running, such as Postgres,Mysql, Kerberos, etc. These tests are not yet + additional services running, such as Postgres, Mysql, Kerberos, etc. These tests are not yet Review comment: ```suggestion additional services running, such as Postgres, Mysql, Kerberos, etc. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7466: [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst
potiuk commented on a change in pull request #7466: [AIRFLOW-] Update tests info in CONTRIBUTING.rst URL: https://github.com/apache/airflow/pull/7466#discussion_r382125829 ## File path: CONTRIBUTING.rst ## @@ -326,13 +326,13 @@ Test Infrastructure We support the following types of tests: -* **Unit tests** are Python ``nose`` tests launched with ``run-tests``. +* **Unit tests** are Python tests launched with ``pytest``. Unit tests are available both in the `Breeze environment `_ and `local virtualenv `_. * **Integration tests** are available in the Breeze development environment that is also used for Airflow Travis CI tests. Integration test are special tests that require - additional services running, such as Postgres,Mysql, Kerberos, etc. These tests are not yet + additional services running, such as Postgres, Mysql, Kerberos, etc. These tests are not yet clearly marked as integration tests but soon they will be clearly separated by the ``pytest`` annotations. Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ANiteckiP edited a comment on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
ANiteckiP edited a comment on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor URL: https://github.com/apache/airflow/pull/7473#issuecomment-589162333 @nuclearpinguin This change would effectively prohibit using poke-only sensors together with SequentialExecutor - SequentialExecutor would force the reschedule mode and the code in PR you linked would cause DAG parsing to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ANiteckiP commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
ANiteckiP commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor URL: https://github.com/apache/airflow/pull/7473#issuecomment-589162333 @nuclearpinguin This change would effectively prohibit using poke-only sensors together with SequentialExecutor - SequentialExecutor would force the reschedule mode and the code you linked would cause DAG parsing to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7410: [DO NOT MERGE] [AIRFLOW-6790] Add basic Tableau Integration
feluelle commented on a change in pull request #7410: [DO NOT MERGE] [AIRFLOW-6790] Add basic Tableau Integration URL: https://github.com/apache/airflow/pull/7410#discussion_r382100602 ## File path: setup.py ## @@ -321,6 +321,7 @@ def write_version(filename: str = os.path.join(*["airflow", "git_version"])): ] salesforce = [ Review comment: @Fokko do you think we should move it to a new provider called Tableau? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7410: [AIRFLOW-6790] Add basic Tableau Integration
feluelle commented on a change in pull request #7410: [AIRFLOW-6790] Add basic
Tableau Integration
URL: https://github.com/apache/airflow/pull/7410#discussion_r382099957
##
File path:
airflow/providers/salesforce/example_dags/example_tableau_refresh_workbook.py
##
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag that performs a refresh operation on a Tableau Workbook
aka Extract. Since this is an
+asynchronous operation we don't know when the operation actually finishes.
That's why we have a second task
+that checks exactly that. So that you can perform further operations after the
extract has been refreshed.
+"""
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.providers.salesforce.operators.tableau_refresh_workbook import
TableauRefreshWorkbookOperator
+from airflow.providers.salesforce.sensors.tableau_job_status import
TableauJobStatusSensor
+from airflow.utils.dates import days_ago
+
+DEFAULT_ARGS = {
+'owner': 'airflow',
+'depends_on_past': False,
+'start_date': days_ago(2),
+'email': ['airf...@example.com'],
+'email_on_failure': False,
+'email_on_retry': False
+}
+
+with DAG(
+dag_id='example_tableau_refresh_workbook',
+default_args=DEFAULT_ARGS,
+dagrun_timeout=timedelta(hours=2),
+schedule_interval=None,
+tags=['example'],
+) as dag:
+task_refresh_workbook = TableauRefreshWorkbookOperator(
+site_id='my_site',
+workbook_name='MyWorkbook',
+task_id='refresh_tableau_workbook',
+dag=dag
+)
+task_check_job_status = TableauJobStatusSensor(
Review comment:
I need to fully test this behaviour first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7410: [AIRFLOW-6790] Add basic Tableau Integration
feluelle commented on a change in pull request #7410: [AIRFLOW-6790] Add basic
Tableau Integration
URL: https://github.com/apache/airflow/pull/7410#discussion_r382099699
##
File path:
airflow/providers/salesforce/example_dags/example_tableau_refresh_workbook.py
##
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag that performs a refresh operation on a Tableau Workbook
aka Extract. Since this is an
+asynchronous operation we don't know when the operation actually finishes.
That's why we have a second task
+that checks exactly that. So that you can perform further operations after the
extract has been refreshed.
+"""
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.providers.salesforce.operators.tableau_refresh_workbook import
TableauRefreshWorkbookOperator
+from airflow.providers.salesforce.sensors.tableau_job_status import
TableauJobStatusSensor
+from airflow.utils.dates import days_ago
+
+DEFAULT_ARGS = {
+'owner': 'airflow',
+'depends_on_past': False,
+'start_date': days_ago(2),
+'email': ['airf...@example.com'],
+'email_on_failure': False,
+'email_on_retry': False
+}
+
+with DAG(
+dag_id='example_tableau_refresh_workbook',
+default_args=DEFAULT_ARGS,
+dagrun_timeout=timedelta(hours=2),
+schedule_interval=None,
+tags=['example'],
+) as dag:
+task_refresh_workbook = TableauRefreshWorkbookOperator(
+site_id='my_site',
+workbook_name='MyWorkbook',
+task_id='refresh_tableau_workbook',
+dag=dag
+)
+task_check_job_status = TableauJobStatusSensor(
Review comment:
I am using the Sensor right know with the default mode `poke` means it will
`sleep` :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8
boring-cyborg[bot] commented on issue #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 URL: https://github.com/apache/airflow/pull/7474#issuecomment-589155404 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] MikaYuoadas opened a new pull request #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8
MikaYuoadas opened a new pull request #7474: [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 URL: https://github.com/apache/airflow/pull/7474 All import from `typing_extensions` should be done through `airflow.typing_compat` to avoid breakage under python >= 3.8 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6854) Missing import `typing_extensions` when running under py38
[
https://issues.apache.org/jira/browse/AIRFLOW-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041106#comment-17041106
]
ASF GitHub Bot commented on AIRFLOW-6854:
-
MikaYuoadas commented on pull request #7474: [AIRFLOW-6854] Fix missing
typing_extensions on python 3.8
URL: https://github.com/apache/airflow/pull/7474
All import from `typing_extensions` should be done through
`airflow.typing_compat` to avoid breakage under python >= 3.8
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Missing import `typing_extensions` when running under py38
> --
>
> Key: AIRFLOW-6854
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6854
> Project: Apache Airflow
> Issue Type: Bug
> Components: build
>Affects Versions: 1.10.9
>Reporter: Akim Sadaoui
>Assignee: Akim Sadaoui
>Priority: Major
>
> Version 1.10.9 (and latest master) crashes when running with python 3.8:
> {code:java}
> File
> "/home/asadaoui/.virtualenvs/airflow/lib/python3.8/site-packages/airflow/serialization/json_schema.py",
> line 26, in
> from typing_extensions import Protocol
> ModuleNotFoundError: No module named 'typing_extensions'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (AIRFLOW-6854) Missing import `typing_extensions` when running under py38
[
https://issues.apache.org/jira/browse/AIRFLOW-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-6854 started by Akim Sadaoui.
-
> Missing import `typing_extensions` when running under py38
> --
>
> Key: AIRFLOW-6854
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6854
> Project: Apache Airflow
> Issue Type: Bug
> Components: build
>Affects Versions: 1.10.9
>Reporter: Akim Sadaoui
>Assignee: Akim Sadaoui
>Priority: Major
>
> Version 1.10.9 (and latest master) crashes when running with python 3.8:
> {code:java}
> File
> "/home/asadaoui/.virtualenvs/airflow/lib/python3.8/site-packages/airflow/serialization/json_schema.py",
> line 26, in
> from typing_extensions import Protocol
> ModuleNotFoundError: No module named 'typing_extensions'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6854) Missing import `typing_extensions` when running under py38
Akim Sadaoui created AIRFLOW-6854:
-
Summary: Missing import `typing_extensions` when running under py38
Key: AIRFLOW-6854
URL: https://issues.apache.org/jira/browse/AIRFLOW-6854
Project: Apache Airflow
Issue Type: Bug
Components: build
Affects Versions: 1.10.9
Reporter: Akim Sadaoui
Assignee: Akim Sadaoui
Version 1.10.9 (and latest master) crashes when running with python 3.8:
{code:java}
File
"/home/asadaoui/.virtualenvs/airflow/lib/python3.8/site-packages/airflow/serialization/json_schema.py",
line 26, in
from typing_extensions import Protocol
ModuleNotFoundError: No module named 'typing_extensions'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041101#comment-17041101 ] Michał Słowikowski commented on AIRFLOW-6732: - Thanks a lot [~toddy86], appreciated it. Have a great day! > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] MichaelChirico commented on issue #7463: [AIRFLOW-XXXX] clarify dag_id parameter
MichaelChirico commented on issue #7463: [AIRFLOW-] clarify dag_id parameter URL: https://github.com/apache/airflow/pull/7463#issuecomment-589153238 failure seems unrelated: >The job exceeded the maximum time limit for jobs, and has been terminated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration
feluelle commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration URL: https://github.com/apache/airflow/pull/7410#discussion_r382087931 ## File path: airflow/providers/salesforce/hooks/tableau.py ## @@ -0,0 +1,110 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from enum import Enum +from typing import Optional + +from tableauserverclient import Pager, PersonalAccessTokenAuth, Server, TableauAuth +from tableauserverclient.server import Auth + +from airflow.hooks.base_hook import BaseHook + + +class TableauJobFinishCode(Enum): Review comment: 😄 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration
Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration URL: https://github.com/apache/airflow/pull/7410#discussion_r382085543 ## File path: airflow/providers/salesforce/hooks/tableau.py ## @@ -0,0 +1,110 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from enum import Enum +from typing import Optional + +from tableauserverclient import Pager, PersonalAccessTokenAuth, Server, TableauAuth +from tableauserverclient.server import Auth + +from airflow.hooks.base_hook import BaseHook + + +class TableauJobFinishCode(Enum): Review comment: 😍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6853) ExternalTaskSensor check_existence does not work with packaged DAGs
[
https://issues.apache.org/jira/browse/AIRFLOW-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes Wienke updated AIRFLOW-6853:
-
Summary: ExternalTaskSensor check_existence does not work with packaged
DAGs (was: ExternalTaskSensor does not work with packaged DAGs)
> ExternalTaskSensor check_existence does not work with packaged DAGs
> ---
>
> Key: AIRFLOW-6853
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6853
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.9
>Reporter: Johannes Wienke
>Priority: Major
>
> The current implementation of ExternalTaskSensor uses the file location of a
> DAG to determine if it still exists. This is realized by the following code
> fragment:
> {code:python}
> if not os.path.exists(dag_to_wait.fileloc):
> raise AirflowException('The external DAG '
>'{} was
> deleted.'.format(self.external_dag_id)
> {code}
> In case the DAG to check is part of a Zip archive, fileloc looks something
> like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
> obviously not be checked with {{os.path.exists}} and therefore the sensor is
> currently unable to handle packaged DAGs at all.
> This also contradicts the explicit note in the DAG model:
> {noformat}
> # The location of the file containing the DAG object
> # Note: Do not depend on fileloc pointing to a file; in the case of a
> # packaged DAG, it will point to the subpath of the DAG within the
> # associated zip.
> {noformat}
> I suspect, the sensor should instead use {{is_active}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration
Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add
basic Tableau Integration
URL: https://github.com/apache/airflow/pull/7410#discussion_r382085093
##
File path:
airflow/providers/salesforce/example_dags/example_tableau_refresh_workbook.py
##
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag that performs a refresh operation on a Tableau Workbook
aka Extract. Since this is an
+asynchronous operation we don't know when the operation actually finishes.
That's why we have a second task
+that checks exactly that. So that you can perform further operations after the
extract has been refreshed.
+"""
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.providers.salesforce.operators.tableau_refresh_workbook import
TableauRefreshWorkbookOperator
+from airflow.providers.salesforce.sensors.tableau_job_status import
TableauJobStatusSensor
+from airflow.utils.dates import days_ago
+
+DEFAULT_ARGS = {
+'owner': 'airflow',
+'depends_on_past': False,
+'start_date': days_ago(2),
+'email': ['airf...@example.com'],
+'email_on_failure': False,
+'email_on_retry': False
+}
+
+with DAG(
+dag_id='example_tableau_refresh_workbook',
+default_args=DEFAULT_ARGS,
+dagrun_timeout=timedelta(hours=2),
+schedule_interval=None,
+tags=['example'],
+) as dag:
+task_refresh_workbook = TableauRefreshWorkbookOperator(
+site_id='my_site',
+workbook_name='MyWorkbook',
+task_id='refresh_tableau_workbook',
+dag=dag
+)
+task_check_job_status = TableauJobStatusSensor(
Review comment:
I would `sleep`, similar to here:
https://github.com/apache/airflow/blob/aff3a361b4092212c0757f9ce88fa2e40d25d1f4/airflow/providers/apache/druid/hooks/druid.py#L99-L124
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #6961: [AIRFLOW-XXX] Add task execution process on Celery Execution diagram
mik-laj opened a new pull request #6961: [AIRFLOW-XXX] Add task execution
process on Celery Execution diagram
URL: https://github.com/apache/airflow/pull/6961
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add basic Tableau Integration
Fokko commented on a change in pull request #7410: [WIP] [AIRFLOW-6790] Add
basic Tableau Integration
URL: https://github.com/apache/airflow/pull/7410#discussion_r382083931
##
File path:
airflow/providers/salesforce/example_dags/example_tableau_refresh_workbook.py
##
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag that performs a refresh operation on a Tableau Workbook
aka Extract. Since this is an
+asynchronous operation we don't know when the operation actually finishes.
That's why we have a second task
+that checks exactly that. So that you can perform further operations after the
extract has been refreshed.
+"""
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.providers.salesforce.operators.tableau_refresh_workbook import
TableauRefreshWorkbookOperator
+from airflow.providers.salesforce.sensors.tableau_job_status import
TableauJobStatusSensor
+from airflow.utils.dates import days_ago
+
+DEFAULT_ARGS = {
+'owner': 'airflow',
+'depends_on_past': False,
+'start_date': days_ago(2),
+'email': ['airf...@example.com'],
+'email_on_failure': False,
+'email_on_retry': False
+}
+
+with DAG(
+dag_id='example_tableau_refresh_workbook',
+default_args=DEFAULT_ARGS,
+dagrun_timeout=timedelta(hours=2),
+schedule_interval=None,
+tags=['example'],
+) as dag:
+task_refresh_workbook = TableauRefreshWorkbookOperator(
Review comment:
I prefer the first one, but I'll leave it up to you :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-6853) ExternalTaskSensor does not work with packaged DAGs
[
https://issues.apache.org/jira/browse/AIRFLOW-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes Wienke updated AIRFLOW-6853:
-
Description:
The current implementation of ExternalTaskSensor uses the file location of a
DAG to determine if it still exists. This is realized by the following code
fragment:
{code:python}
if not os.path.exists(dag_to_wait.fileloc):
raise AirflowException('The external DAG '
'{} was
deleted.'.format(self.external_dag_id)
{code}
In case the DAG to check is part of a Zip archive, fileloc looks something
like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
obviously not be checked with {{os.path.exists}} and therefore the sensor is
currently unable to handle packaged DAGs at all.
This also contradicts the explicit note in the DAG model:
{noformat}
# The location of the file containing the DAG object
# Note: Do not depend on fileloc pointing to a file; in the case of a
# packaged DAG, it will point to the subpath of the DAG within the
# associated zip.
{noformat}
I suspect, the sensor should instead use {{is_active}}.
was:
The current implementation of ExternalTaskSensor uses the file location of a
DAG to determine if it still exists. This is realized by the following code
fragment:
{code:python}
if not os.path.exists(dag_to_wait.fileloc):
raise AirflowException('The external DAG '
'{} was
deleted.'.format(self.external_dag_id)
{code}
In case the DAG to check is part of a Zip archive, fileloc looks something
like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
obviously not be checked with {{os.path.exists}} and therefore the sensor is
currently unable to handle packaged DAGs at all.
> ExternalTaskSensor does not work with packaged DAGs
> ---
>
> Key: AIRFLOW-6853
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6853
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.9
>Reporter: Johannes Wienke
>Priority: Major
>
> The current implementation of ExternalTaskSensor uses the file location of a
> DAG to determine if it still exists. This is realized by the following code
> fragment:
> {code:python}
> if not os.path.exists(dag_to_wait.fileloc):
> raise AirflowException('The external DAG '
>'{} was
> deleted.'.format(self.external_dag_id)
> {code}
> In case the DAG to check is part of a Zip archive, fileloc looks something
> like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
> obviously not be checked with {{os.path.exists}} and therefore the sensor is
> currently unable to handle packaged DAGs at all.
> This also contradicts the explicit note in the DAG model:
> {noformat}
> # The location of the file containing the DAG object
> # Note: Do not depend on fileloc pointing to a file; in the case of a
> # packaged DAG, it will point to the subpath of the DAG within the
> # associated zip.
> {noformat}
> I suspect, the sensor should instead use {{is_active}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6853) ExternalTaskSensor does not work with packaged DAGs
Johannes Wienke created AIRFLOW-6853:
Summary: ExternalTaskSensor does not work with packaged DAGs
Key: AIRFLOW-6853
URL: https://issues.apache.org/jira/browse/AIRFLOW-6853
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.9
Reporter: Johannes Wienke
The current implementation of ExternalTaskSensor uses the file location of a
DAG to determine if it still exists. This is realized by the following code
fragment:
{code:python}
if not os.path.exists(dag_to_wait.fileloc):
raise AirflowException('The external DAG '
'{} was
deleted.'.format(self.external_dag_id)
{code}
In case the DAG to check is part of a Zip archive, fileloc looks something
like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can
obviously not be checked with {{os.path.exists}} and therefore the sensor is
currently unable to handle packaged DAGs at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041078#comment-17041078 ] Todd de Quincey edited comment on AIRFLOW-6732 at 2/20/20 3:30 PM: --- Hi [~mislow], I have created the following: * GoogleAdsHook - supports connecting to google ads via a service account using the yaml config and access to the search method from an authenticated service. The search method accepts an arbitrary Google Ads Query Language query and returns a list of Google Ads Row objects. See the following link for reference (not exactly the same, but similar use of the service and search methods) [https://developers.google.com/google-ads/api/docs/samples/get-campaigns] * GoogleAdsToGcsOperator - Takes a list of client_ids and a Google Ads Query Language query, gets the results from the API, converts them to a CSV and uploads them to a designated GCS bucket. FYI, the links you have provided above are for the Google Ad Words API. As I understand it, the Google Ads API (still in beta) is being developed to replace this API and the functionality of the new Google Ads API (at the date I created the hook and operator) wasn't comparable to the old API. was (Author: toddy86): Hi [~mislow], I have created the following: * GoogleAdsHook - supports connecting to google ads via a service account using the yaml config and access to the search method from an authenticated service. The search method accepts an arbitrary Google Ads Query Language query and returns a list of Google Ads Row objects. See the following link for reference (not exactly the same, but similar use of the service and search methods) [https://developers.google.com/google-ads/api/docs/samples/get-campaigns] * GoogleAdsToGcsOperator - Takes a list of client_ids and a Google Ads Query Language query, gets the results from the API, converts them to a CSV and uploads them to a designated GCS bucket. FYI, the links you have provided above are for the Google Ad Words API. As I understand it, the Google Ads API (still in beta) is being developed to replace this API. > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041080#comment-17041080 ] Todd de Quincey commented on AIRFLOW-6732: -- [~turbaszek], I will be taking you up on your kind offer of help! > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041078#comment-17041078 ] Todd de Quincey commented on AIRFLOW-6732: -- Hi [~mislow], I have created the following: * GoogleAdsHook - supports connecting to google ads via a service account using the yaml config and access to the search method from an authenticated service. The search method accepts an arbitrary Google Ads Query Language query and returns a list of Google Ads Row objects. See the following link for reference (not exactly the same, but similar use of the service and search methods) [https://developers.google.com/google-ads/api/docs/samples/get-campaigns] * GoogleAdsToGcsOperator - Takes a list of client_ids and a Google Ads Query Language query, gets the results from the API, converts them to a CSV and uploads them to a designated GCS bucket. FYI, the links you have provided above are for the Google Ad Words API. As I understand it, the Google Ads API (still in beta) is being developed to replace this API. > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #6961: [AIRFLOW-XXX] Add task execution process on Celery Execution diagram
stale[bot] closed pull request #6961: [AIRFLOW-XXX] Add task execution process on Celery Execution diagram URL: https://github.com/apache/airflow/pull/6961 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
nuclearpinguin commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor URL: https://github.com/apache/airflow/pull/7473#issuecomment-589096888 Can you check how this will work with the following change? https://github.com/apache/airflow/pull/7197 Stateful sensors should not be used in reschedule mode. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6852) Force rescheule mode for sensors when running on SequentialExecutor to avoid deadlocks.
[ https://issues.apache.org/jira/browse/AIRFLOW-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041057#comment-17041057 ] ASF GitHub Bot commented on AIRFLOW-6852: - ANiteckiP commented on pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor URL: https://github.com/apache/airflow/pull/7473 Force reschedule mode for sensors when running on SequentialExecutor to avoid deadlocks. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Force rescheule mode for sensors when running on SequentialExecutor to avoid > deadlocks. > --- > > Key: AIRFLOW-6852 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6852 > Project: Apache Airflow > Issue Type: Bug > Components: executors >Affects Versions: 2.0.0 >Reporter: Aleksander Nitecki >Assignee: Aleksander Nitecki >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ANiteckiP opened a new pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
ANiteckiP opened a new pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor URL: https://github.com/apache/airflow/pull/7473 Force reschedule mode for sensors when running on SequentialExecutor to avoid deadlocks. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6558) Campaign Manager Operators to insert and modify conversions
[ https://issues.apache.org/jira/browse/AIRFLOW-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041055#comment-17041055 ] ASF GitHub Bot commented on AIRFLOW-6558: - potiuk commented on pull request #7420: [AIRFLOW-6558] Campaign Manager operators for conversions URL: https://github.com/apache/airflow/pull/7420 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Campaign Manager Operators to insert and modify conversions > --- > > Key: AIRFLOW-6558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6558 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6558) Campaign Manager Operators to insert and modify conversions
[ https://issues.apache.org/jira/browse/AIRFLOW-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6558. --- Fix Version/s: 2.0.0 Resolution: Fixed > Campaign Manager Operators to insert and modify conversions > --- > > Key: AIRFLOW-6558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6558 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6558) Campaign Manager Operators to insert and modify conversions
[ https://issues.apache.org/jira/browse/AIRFLOW-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041056#comment-17041056 ] ASF subversion and git services commented on AIRFLOW-6558: -- Commit aff3a361b4092212c0757f9ce88fa2e40d25d1f4 in airflow's branch refs/heads/master from Tomek Urbaszek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=aff3a36 ] [AIRFLOW-6558] Campaign Manager operators for conversions (#7420) > Campaign Manager Operators to insert and modify conversions > --- > > Key: AIRFLOW-6558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6558 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7420: [AIRFLOW-6558] Campaign Manager operators for conversions
potiuk merged pull request #7420: [AIRFLOW-6558] Campaign Manager operators for conversions URL: https://github.com/apache/airflow/pull/7420 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-589092958 @digger is right, Dataproc was not working (tested using example DAG). I've tried to do my best to preserve as much backward compatibility as I could. The present implementation is much clearer and uses Google's python library. Without operators - hooks coupling it should be easier to add any needed changes. I am happy to review PR that adds missing functionality @dossett :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6852) Force rescheule mode for sensors when running on SequentialExecutor to avoid deadlocks.
Aleksander Nitecki created AIRFLOW-6852: --- Summary: Force rescheule mode for sensors when running on SequentialExecutor to avoid deadlocks. Key: AIRFLOW-6852 URL: https://issues.apache.org/jira/browse/AIRFLOW-6852 Project: Apache Airflow Issue Type: Bug Components: executors Affects Versions: 2.0.0 Reporter: Aleksander Nitecki Assignee: Aleksander Nitecki -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze
potiuk commented on issue #7458: [AIRFLOW-6838][WIP] Introduce real subcommands for Breeze URL: https://github.com/apache/airflow/pull/7458#issuecomment-589085051 Will do ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041010#comment-17041010 ] Michał Słowikowski edited comment on AIRFLOW-6732 at 2/20/20 2:25 PM: -- Hi [~toddy86]. Could list hooks and operators are you going to implement? I am asking because I want to create operator for GA360AdsLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/list]), GA360ReportingLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/get]) and GA360ReportingLinksRetrieveOperator, but I do not have API for it. was (Author: mislow): Hi [~toddy86]. Could list hooks and operators are you going to implement? I am asking because I want to create operator for GA360AdsLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/list]), GA360ReportingLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/get]) and GA360ReportingLinksRetrieveOperator, but I have API for it. > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041010#comment-17041010 ] Michał Słowikowski edited comment on AIRFLOW-6732 at 2/20/20 2:24 PM: -- Hi [~toddy86]. Could list hooks and operators are you going to implement? I am asking because I want to create operator for GA360AdsLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/list]), GA360ReportingLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/get]) and GA360ReportingLinksRetrieveOperator, but I have API for it. was (Author: mislow): Hi [~toddy86]. Could list hooks and operators are you going to implement? I am asking because I want to create operator for GA360AdsLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/list]) and GA360ReportingLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/get]). > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
