[jira] [Updated] (AIRFLOW-5702) Fix common docstring issues
[ https://issues.apache.org/jira/browse/AIRFLOW-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-5702: --- Fix Version/s: (was: 2.0.0) 1.10.7 > Fix common docstring issues > --- > > Key: AIRFLOW-5702 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5702 > Project: Apache Airflow > Issue Type: Bug > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5702) Fix common docstring issues
[ https://issues.apache.org/jira/browse/AIRFLOW-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955397#comment-16955397 ] ASF subversion and git services commented on AIRFLOW-5702: -- Commit 4903c9730c09f8a98bdf1d891479be0b1cd238c8 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4903c97 ] [AIRFLOW-5702] Fix common docstring issues (#6372) > Fix common docstring issues > --- > > Key: AIRFLOW-5702 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5702 > Project: Apache Airflow > Issue Type: Bug > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5702) Fix common docstring issues
[ https://issues.apache.org/jira/browse/AIRFLOW-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955395#comment-16955395 ] ASF GitHub Bot commented on AIRFLOW-5702: - mik-laj commented on pull request #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix common docstring issues > --- > > Key: AIRFLOW-5702 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5702 > Project: Apache Airflow > Issue Type: Bug > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup
codecov-io edited a comment on issue #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup URL: https://github.com/apache/airflow/pull/6373#issuecomment-544193294 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=h1) Report > Merging [#6373](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c72c42730236fee1526fcc03dca7f88e1778ee94?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6373 +/- ## == - Coverage80.4% 80.08% -0.33% == Files 616 616 Lines 3580535805 == - Hits2879028674 -116 - Misses 7015 7131 +116 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `33.33% <0%> (-41.67%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.14% <0%> (-28.36%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.56% <0%> (-0.17%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=footer). Last update [c72c427...4bd3831](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6372: [AIRFLOW-5702] Fix common docstring issues
mik-laj merged pull request #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup
codecov-io edited a comment on issue #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup URL: https://github.com/apache/airflow/pull/6373#issuecomment-544193294 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=h1) Report > Merging [#6373](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c72c42730236fee1526fcc03dca7f88e1778ee94?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6373 +/- ## == - Coverage80.4% 80.08% -0.33% == Files 616 616 Lines 3580535805 == - Hits2879028674 -116 - Misses 7015 7131 +116 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `33.33% <0%> (-41.67%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.14% <0%> (-28.36%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.56% <0%> (-0.17%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=footer). Last update [c72c427...4bd3831](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-544220915 @mik-laj interesting. It's related but different yeah? Looks like that one is about providing support for alternative to fernet. Is that right? Here I am trying to provide a means to source connections from arbitrary creds server, i.e. other than env vars / metastore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish edited a comment on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-544220915 @mik-laj interesting. It's related but different yeah? Looks like that one is about providing support for alternative to fernet. Is that right? Here I am providing means to source connections from other arbitrary creds server, i.e. other than env vars / metastore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup
potiuk commented on a change in pull request #6373: [AIRFLOW-5704] Optimise
KIND kubernetes setup
URL: https://github.com/apache/airflow/pull/6373#discussion_r336760858
##
File path: Dockerfile
##
@@ -174,39 +170,8 @@ RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci"
]]; then \
;\
fi
-# TODO: We should think about removing those and moving them into
docker-compose dependencies.
COPY scripts/ci/docker_build/ci_build_install_deps.sh
/tmp/ci_build_install_deps.sh
-# Kubernetes dependencies
-RUN \
-if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
Review comment:
It's not at runtime - it's just grouping all the CI-related installs in
single RUN command.
This is mostly to decrease the build time for slim (in the future
production) image. The problem is that currently in multi-staging images, even
if you have a stage that is not used, it will not be skipped (this will be
improved when BUILDKIT goes out of beta and we will be able to use it including
caching - in BUILDKIT, unused stages are skipped while building). The "if"
there is a hack to skip building CI dependencies if we are only building slim
image.
Separating it to single script makes it slightly faster to rebuild the slim
image- every RUN command takes couple of seconds when executed so if we have a
separate RUN and if in every RUN, we got 20-30 seconds longer builds always
(because this stage is not part of the image, so we do not have cache for it,
it is also not cached so it is always built - even if we locally built the slim
image. Unfortunately, we cannot use both --cache-from and local cache at the
same time (https://github.com/moby/moby/issues/32612) - this could also solve
the problem.
This will be reverted back to RUN commands in the PROD image change I am
working on #6266 where I got rid of the slim image.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6373: [AIRFLOW-5704] Optimise KIND kubernetes setup
potiuk commented on a change in pull request #6373: [AIRFLOW-5704] Optimise
KIND kubernetes setup
URL: https://github.com/apache/airflow/pull/6373#discussion_r336760858
##
File path: Dockerfile
##
@@ -174,39 +170,8 @@ RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci"
]]; then \
;\
fi
-# TODO: We should think about removing those and moving them into
docker-compose dependencies.
COPY scripts/ci/docker_build/ci_build_install_deps.sh
/tmp/ci_build_install_deps.sh
-# Kubernetes dependencies
-RUN \
-if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
Review comment:
It's not at runtime - it's just grouping all the CI-related installs in
single RUN command.
This is mostly to decrease the build time for slim (in the future
production) image. The problem is that currently in multi-staging images, even
if you have a stage that is not used, it will not be skipped (this will be
improved when BUILDKIT goes out of beta and we will be able to use it including
caching - in BUILDKIT, unused stages are skipped while building). The "if"
there is a hack to skip building CI dependencies if we are only building slim
image.
Separating it to single script makes it slightly faster to rebuild the slim
image- every RUN command takes couple of seconds when executed so if we have a
separate RUN and if in every RUN, we got 20-30 seconds longer builds always
(because this stage is not part of the image, so we do not have cache for it,
it is also not cached so it is always built - even if we locally built the slim
image. Unfortunately, we cannot use both --cache-from and local cache at the
same time (https://github.com/moby/moby/issues/32612) - this could also solve
the problem.
This will be reverted back to RUN commands in the PROD image change I am
working on #6266
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5705) add option for alternative creds backend
[ https://issues.apache.org/jira/browse/AIRFLOW-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Standish updated AIRFLOW-5705: - Description: Idea here is to create some kind of generic creds backend that could support using other creds stores such as AWS SSM parameter store. was: Idea hear is to create some kind of generic creds backend that could support using other creds stores such as AWS SSM parameter store. > add option for alternative creds backend > > > Key: AIRFLOW-5705 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5705 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Affects Versions: 1.10.5 >Reporter: Daniel Standish >Assignee: Daniel Standish >Priority: Major > > Idea here is to create some kind of generic creds backend that could support > using other creds stores such as AWS SSM parameter store. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dstandish commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-544220915 @mik-laj interesting. It's related but different yeah? Looks like that one is about providing support for alternative to fernet. Here I am providing means to source connections from places other than env vars / metastore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
codecov-io edited a comment on issue #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator URL: https://github.com/apache/airflow/pull/6259#issuecomment-538457664 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=h1) Report > Merging [#6259](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6259 +/- ## == + Coverage 80.05% 80.07% +0.01% == Files 610 611 +1 Lines 3526435270 +6 == + Hits2823228241 +9 + Misses 7032 7029 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/example\_dags/example\_bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9iaWdxdWVyeV90b19iaWdxdWVyeS5weQ==) | `100% <100%> (ø)` | | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.38% <0%> (+0.2%)` | :arrow_up: | | [airflow/operators/bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmlncXVlcnlfdG9fYmlncXVlcnkucHk=) | `100% <0%> (+6.45%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=footer). Last update [76fe45e...f23de31](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages
mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages URL: https://github.com/apache/airflow/pull/6364#discussion_r336762803 ## File path: airflow/contrib/hooks/slack_webhook_hook.py ## @@ -42,6 +42,9 @@ class SlackWebhookHook(HttpHook): :param attachments: The attachments to send on Slack. Should be a list of dictionaries representing Slack attachments. :type attachments: list +:param blocks: The blocks to send on Slack. Should be a list of + dictionaries representing Slack blocks. Review comment: It was intended to align the columns for the second line with the first description line. with 3 spaces (current) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` with 4 spaces (suggestion) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. # or :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` But, there is already 3 space-indented line. (20 + 3 spaces) > I think this is also for aligning the columns. ``` :param link_names: Whether or not to find and link channel and usernames in your message ``` It is a little confusing for me. Which rules should I follow? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages
mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages URL: https://github.com/apache/airflow/pull/6364#discussion_r336762803 ## File path: airflow/contrib/hooks/slack_webhook_hook.py ## @@ -42,6 +42,9 @@ class SlackWebhookHook(HttpHook): :param attachments: The attachments to send on Slack. Should be a list of dictionaries representing Slack attachments. :type attachments: list +:param blocks: The blocks to send on Slack. Should be a list of + dictionaries representing Slack blocks. Review comment: It was intended to align the columns for the second line with the first description line. with 3 spaces (current) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` with 4 spaces (suggestion) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. # or :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` And, there is already 3 space-indented line. (20 + 3 spaces) > I think this is also for aligning the columns. ``` :param link_names: Whether or not to find and link channel and usernames in your message ``` It is a little confusing for me. Which rules should I follow? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages
mingrammer commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages URL: https://github.com/apache/airflow/pull/6364#discussion_r336762803 ## File path: airflow/contrib/hooks/slack_webhook_hook.py ## @@ -42,6 +42,9 @@ class SlackWebhookHook(HttpHook): :param attachments: The attachments to send on Slack. Should be a list of dictionaries representing Slack attachments. :type attachments: list +:param blocks: The blocks to send on Slack. Should be a list of + dictionaries representing Slack blocks. Review comment: It was intended to align the columns for the second line with the first description line. with 3 spaces (current) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` with 4 spaces (suggestion) ```python :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. # or :param blocks: The blocks to send on Slack. Should be a list of dictionaries representing Slack blocks. ``` And, there is already 3 space-indented line. (20 + 3 spaces) ``` :param link_names: Whether or not to find and link channel and usernames in your message ``` It is a little confusing for me. Which rules should I follow? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5684) Docker-compose-kubernetes still used by breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5684. --- Fix Version/s: 1.10.7 Resolution: Fixed > Docker-compose-kubernetes still used by breeze > -- > > Key: AIRFLOW-5684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5684 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > After transitioning to KinD for kubernetes testing, we no longer need the > docker-compose-kubernetes.yaml. However it still looks for this > docker-compose in the breeze command. This PR removes these unneeded commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5223) Use KinD for Kubernetes in CI
[ https://issues.apache.org/jira/browse/AIRFLOW-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5223. --- Fix Version/s: (was: 2.0.0) 1.10.7 Resolution: Fixed > Use KinD for Kubernetes in CI > - > > Key: AIRFLOW-5223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5223 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.4 >Reporter: Gerardo Curiel >Priority: Minor > Fix For: 1.10.7 > > > This PR reimplements Kubernetes integration testing using kind, a tool for > running local Kubernetes clusters using Docker container "nodes". The "nodes" > are deployed inside the main airflow-testing container, so I needed to also > install docker, hence, this particular implementation relies on > docker-in-docker. > For completeness sake, I considered mounting the host's /var/run/docker.sock > socket so that we could rely on the host's Docker daemon instead of running > docker-in-docker, but I felt docker-in-docker, being a more self-contained > implementation, offers the least amount of friction when using it locally for > development purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5649) Skipping tests for non relevant changes
[ https://issues.apache.org/jira/browse/AIRFLOW-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5649: -- Affects Version/s: 1.10.6 > Skipping tests for non relevant changes > --- > > Key: AIRFLOW-5649 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5649 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > We can skip running tests for some changes that are not relevant: > All the tests will always run for direct pushes. This will run all tests in > master branch and v1-10-tests always (no matter if they are cron job or not). > For PRs it works like that: > * if there are no changes in .py files -> the separate pylint check will be > skipped altogether (pre-commit will skip the checks anyway in this case but > we save time to pull/build images). > * if there is no change in .py .js .html . all unit tests will be skipped > * if there is a change in .py .js .html files but no kubernetes/* nor > tests/kubernetes/* nor airflow/www/.* the Kubernetes unit tests will be > skipped > We still run all test in CRON on master so in case of any false negatives we > will catch them anyway. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5649) Skipping tests for non relevant changes
[ https://issues.apache.org/jira/browse/AIRFLOW-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5649: -- Fix Version/s: (was: 2.0.0) 1.10.7 > Skipping tests for non relevant changes > --- > > Key: AIRFLOW-5649 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5649 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > We can skip running tests for some changes that are not relevant: > All the tests will always run for direct pushes. This will run all tests in > master branch and v1-10-tests always (no matter if they are cron job or not). > For PRs it works like that: > * if there are no changes in .py files -> the separate pylint check will be > skipped altogether (pre-commit will skip the checks anyway in this case but > we save time to pull/build images). > * if there is no change in .py .js .html . all unit tests will be skipped > * if there is a change in .py .js .html files but no kubernetes/* nor > tests/kubernetes/* nor airflow/www/.* the Kubernetes unit tests will be > skipped > We still run all test in CRON on master so in case of any false negatives we > will catch them anyway. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5694) Check for blinker when detecting Sentry packages
[
https://issues.apache.org/jira/browse/AIRFLOW-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-5694.
---
Fix Version/s: (was: 1.10.6)
1.10.7
Resolution: Fixed
> Check for blinker when detecting Sentry packages
>
>
> Key: AIRFLOW-5694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5694
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Minor
> Fix For: 1.10.7
>
>
> After upgrading to 1.10.6rc1 with `sentry-sdk` installed but not specifying
> the `[sentry]` extra, the dependency `blinker` will cause failures of the
> following form:
> {code:python}
> ../lib/python3.7/site-packages/airflow/__init__.py:40: in
> from airflow.models import DAG
> ../lib/python3.7/site-packages/airflow/models/__init__.py:21: in
> from airflow.models.baseoperator import BaseOperator # noqa: F401
> ../lib/python3.7/site-packages/airflow/models/baseoperator.py:42: in
> from airflow.models.dag import DAG
> ../lib/python3.7/site-packages/airflow/models/dag.py:51: in
> from airflow.models.taskinstance import TaskInstance, clear_task_instances
> ../lib/python3.7/site-packages/airflow/models/taskinstance.py:53: in
> from airflow.sentry import Sentry
> ../lib/python3.7/site-packages/airflow/sentry.py:167: in
> Sentry = ConfiguredSentry()
> ../lib/python3.7/site-packages/airflow/sentry.py:94: in __init__
> init(integrations=integrations)
> ../lib/python3.7/site-packages/sentry_sdk/hub.py:81: in _init
> client = Client(*args, **kwargs) # type: ignore
> ../lib/python3.7/site-packages/sentry_sdk/client.py:80: in __init__
> self._init_impl()
> ../lib/python3.7/site-packages/sentry_sdk/client.py:108: in _init_impl
> with_defaults=self.options["default_integrations"],
> ../lib/python3.7/site-packages/sentry_sdk/integrations/__init__.py:82: in
> setup_integrations
> type(integration).setup_once()
> ../lib/python3.7/site-packages/sentry_sdk/integrations/flask.py:57: in
> setup_once
> appcontext_pushed.connect(_push_appctx)
> ../lib/python3.7/site-packages/flask/signals.py:39: in _fail
> "Signalling support is unavailable because the blinker"
> E RuntimeError: Signalling support is unavailable because the blinker
> library is not installed.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5694) Check for blinker when detecting Sentry packages
[
https://issues.apache.org/jira/browse/AIRFLOW-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955387#comment-16955387
]
ASF subversion and git services commented on AIRFLOW-5694:
--
Commit 288afb5820436a8770ef73bc133dbd617cbfb94c in airflow's branch
refs/heads/v1-10-test from Marcus Levine
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=288afb5 ]
[AIRFLOW-5694] Check for blinker in Sentry setup (#6365)
(cherry picked from commit c72c42730236fee1526fcc03dca7f88e1778ee94)
> Check for blinker when detecting Sentry packages
>
>
> Key: AIRFLOW-5694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5694
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Minor
> Fix For: 1.10.6
>
>
> After upgrading to 1.10.6rc1 with `sentry-sdk` installed but not specifying
> the `[sentry]` extra, the dependency `blinker` will cause failures of the
> following form:
> {code:python}
> ../lib/python3.7/site-packages/airflow/__init__.py:40: in
> from airflow.models import DAG
> ../lib/python3.7/site-packages/airflow/models/__init__.py:21: in
> from airflow.models.baseoperator import BaseOperator # noqa: F401
> ../lib/python3.7/site-packages/airflow/models/baseoperator.py:42: in
> from airflow.models.dag import DAG
> ../lib/python3.7/site-packages/airflow/models/dag.py:51: in
> from airflow.models.taskinstance import TaskInstance, clear_task_instances
> ../lib/python3.7/site-packages/airflow/models/taskinstance.py:53: in
> from airflow.sentry import Sentry
> ../lib/python3.7/site-packages/airflow/sentry.py:167: in
> Sentry = ConfiguredSentry()
> ../lib/python3.7/site-packages/airflow/sentry.py:94: in __init__
> init(integrations=integrations)
> ../lib/python3.7/site-packages/sentry_sdk/hub.py:81: in _init
> client = Client(*args, **kwargs) # type: ignore
> ../lib/python3.7/site-packages/sentry_sdk/client.py:80: in __init__
> self._init_impl()
> ../lib/python3.7/site-packages/sentry_sdk/client.py:108: in _init_impl
> with_defaults=self.options["default_integrations"],
> ../lib/python3.7/site-packages/sentry_sdk/integrations/__init__.py:82: in
> setup_integrations
> type(integration).setup_once()
> ../lib/python3.7/site-packages/sentry_sdk/integrations/flask.py:57: in
> setup_once
> appcontext_pushed.connect(_push_appctx)
> ../lib/python3.7/site-packages/flask/signals.py:39: in _fail
> "Signalling support is unavailable because the blinker"
> E RuntimeError: Signalling support is unavailable because the blinker
> library is not installed.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5680: -- Fix Version/s: (was: 2.0.0) 1.10.7 > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955386#comment-16955386 ] ASF subversion and git services commented on AIRFLOW-5680: -- Commit 2d6e31ff8d5bb8122a29671406207e7a217c80e8 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2d6e31f ] [AIRFLOW-5680] Fixes Kubernetes hangs (#6347) (cherry picked from commit c19a60c23df9dd4fab42e68e9a1992f7f89c281c) > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955384#comment-16955384 ] ASF subversion and git services commented on AIRFLOW-5680: -- Commit 9c63b613736678e018ef047fe7e2193f84be3f01 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9c63b61 ] [AIRFLOW-5680] Fixes Kubernetes hangs (#6347) (cherry picked from commit c19a60c23df9dd4fab42e68e9a1992f7f89c281c) > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5223) Use KinD for Kubernetes in CI
[ https://issues.apache.org/jira/browse/AIRFLOW-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955382#comment-16955382 ] ASF subversion and git services commented on AIRFLOW-5223: -- Commit feef844e7d6cf77d76f17d11b5b5d517cba8fc0b in airflow's branch refs/heads/v1-10-test from Gerardo Curiel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=feef844 ] [AIRFLOW-5223] Use kind for Kubernetes in CI (#5837) This PR reimplements Kubernetes integration testing using kind, a tool for running local Kubernetes clusters using Docker container "nodes". The "nodes" are deployed to a separate docker daemon (dind) started through docker-compose. (cherry picked from commit e62056b2254e1447e20179f8d2911a93462cddc5) > Use KinD for Kubernetes in CI > - > > Key: AIRFLOW-5223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5223 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.4 >Reporter: Gerardo Curiel >Priority: Minor > Fix For: 2.0.0 > > > This PR reimplements Kubernetes integration testing using kind, a tool for > running local Kubernetes clusters using Docker container "nodes". The "nodes" > are deployed inside the main airflow-testing container, so I needed to also > install docker, hence, this particular implementation relies on > docker-in-docker. > For completeness sake, I considered mounting the host's /var/run/docker.sock > socket so that we could rely on the host's Docker daemon instead of running > docker-in-docker, but I felt docker-in-docker, being a more self-contained > implementation, offers the least amount of friction when using it locally for > development purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5649) Skipping tests for non relevant changes
[ https://issues.apache.org/jira/browse/AIRFLOW-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955383#comment-16955383 ] ASF subversion and git services commented on AIRFLOW-5649: -- Commit 68413ac84dfd2285c5e86d51d948c0517c9493f2 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=68413ac ] [AIRFLOW-5649] Skips tests when relevant .py files are not changed (#6321) (cherry picked from commit 10c3f7f0eda199355d32165c148e714459d51c2e) > Skipping tests for non relevant changes > --- > > Key: AIRFLOW-5649 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5649 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > > We can skip running tests for some changes that are not relevant: > All the tests will always run for direct pushes. This will run all tests in > master branch and v1-10-tests always (no matter if they are cron job or not). > For PRs it works like that: > * if there are no changes in .py files -> the separate pylint check will be > skipped altogether (pre-commit will skip the checks anyway in this case but > we save time to pull/build images). > * if there is no change in .py .js .html . all unit tests will be skipped > * if there is a change in .py .js .html files but no kubernetes/* nor > tests/kubernetes/* nor airflow/www/.* the Kubernetes unit tests will be > skipped > We still run all test in CRON on master so in case of any false negatives we > will catch them anyway. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5694) Check for blinker when detecting Sentry packages
[
https://issues.apache.org/jira/browse/AIRFLOW-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955385#comment-16955385
]
ASF subversion and git services commented on AIRFLOW-5694:
--
Commit 708d16ecd1c82c5f26a47712f239d1077eb9fc35 in airflow's branch
refs/heads/v1-10-test from Marcus Levine
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=708d16e ]
[AIRFLOW-5694] Check for blinker in Sentry setup (#6365)
(cherry picked from commit c72c42730236fee1526fcc03dca7f88e1778ee94)
> Check for blinker when detecting Sentry packages
>
>
> Key: AIRFLOW-5694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5694
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Minor
> Fix For: 1.10.6
>
>
> After upgrading to 1.10.6rc1 with `sentry-sdk` installed but not specifying
> the `[sentry]` extra, the dependency `blinker` will cause failures of the
> following form:
> {code:python}
> ../lib/python3.7/site-packages/airflow/__init__.py:40: in
> from airflow.models import DAG
> ../lib/python3.7/site-packages/airflow/models/__init__.py:21: in
> from airflow.models.baseoperator import BaseOperator # noqa: F401
> ../lib/python3.7/site-packages/airflow/models/baseoperator.py:42: in
> from airflow.models.dag import DAG
> ../lib/python3.7/site-packages/airflow/models/dag.py:51: in
> from airflow.models.taskinstance import TaskInstance, clear_task_instances
> ../lib/python3.7/site-packages/airflow/models/taskinstance.py:53: in
> from airflow.sentry import Sentry
> ../lib/python3.7/site-packages/airflow/sentry.py:167: in
> Sentry = ConfiguredSentry()
> ../lib/python3.7/site-packages/airflow/sentry.py:94: in __init__
> init(integrations=integrations)
> ../lib/python3.7/site-packages/sentry_sdk/hub.py:81: in _init
> client = Client(*args, **kwargs) # type: ignore
> ../lib/python3.7/site-packages/sentry_sdk/client.py:80: in __init__
> self._init_impl()
> ../lib/python3.7/site-packages/sentry_sdk/client.py:108: in _init_impl
> with_defaults=self.options["default_integrations"],
> ../lib/python3.7/site-packages/sentry_sdk/integrations/__init__.py:82: in
> setup_integrations
> type(integration).setup_once()
> ../lib/python3.7/site-packages/sentry_sdk/integrations/flask.py:57: in
> setup_once
> appcontext_pushed.connect(_push_appctx)
> ../lib/python3.7/site-packages/flask/signals.py:39: in _fail
> "Signalling support is unavailable because the blinker"
> E RuntimeError: Signalling support is unavailable because the blinker
> library is not installed.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5684) Docker-compose-kubernetes still used by breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955381#comment-16955381 ] ASF subversion and git services commented on AIRFLOW-5684: -- Commit 7961026d5f89b5f633c25f90dc1b552ce6400d63 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7961026 ] [AIRFLOW-5684] docker-compose-kubernetes still used (#6353) (cherry picked from commit f8794d888fcb032fd620a27c1c54312145e9ffb6) > Docker-compose-kubernetes still used by breeze > -- > > Key: AIRFLOW-5684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5684 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After transitioning to KinD for kubernetes testing, we no longer need the > docker-compose-kubernetes.yaml. However it still looks for this > docker-compose in the breeze command. This PR removes these unneeded commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5705) add option for alternative creds backend
[
https://issues.apache.org/jira/browse/AIRFLOW-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955377#comment-16955377
]
ASF GitHub Bot commented on AIRFLOW-5705:
-
dstandish commented on pull request #6376: [WIP] [AIRFLOW-5705] Add creds
backend classes including AWS SSM
URL: https://github.com/apache/airflow/pull/6376
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> add option for alternative creds backend
>
>
> Key: AIRFLOW-5705
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5705
> Project: Apache Airflow
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.10.5
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
>
> Idea hear is to create some kind of generic creds backend that could support
> using other creds stores such as AWS SSM parameter store.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
mik-laj commented on issue #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM URL: https://github.com/apache/airflow/pull/6376#issuecomment-544219078 Work on another key encryption mechanism is available here. https://github.com/jakahn/incubator-airflow/pulls This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish opened a new pull request #6376: [WIP] [AIRFLOW-5705] Add creds backend classes including AWS SSM
dstandish opened a new pull request #6376: [WIP] [AIRFLOW-5705] Add creds
backend classes including AWS SSM
URL: https://github.com/apache/airflow/pull/6376
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#discussion_r336761021 ## File path: .travis.yml ## @@ -63,8 +43,28 @@ jobs: - name: "Build documentation" stage: pre-test script: ./scripts/ci/ci_docs.sh +- name: "Tests postgres kubernetes python 3.6 (persistent)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=persistent_mode + python: "3.6" + stage: test +- name: "Tests postgres kubernetes python 3.6 (git)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=git_mode + python: "3.6" + stage: test +- name: "Tests postgres python 3.6" + env: BACKEND=postgres ENV=docker + python: "3.6" + stage: test +- name: "Tests sqlite python 3.5" + env: BACKEND=sqlite ENV=docker + python: "3.5" + stage: test +- name: "Tests mysql python 3.7" + env: BACKEND=mysql ENV=docker + python: "3.7" + stage: test services: - docker before_install: - ./scripts/ci/ci_before_install.sh -script: "./scripts/ci/ci_run_airflow_testing.sh" +script: travis_wait 30 "./scripts/ci/ci_run_airflow_testing.sh" Review comment: Actually in this change I am going to optimise it a bit more and will only enable travis_wait for kubernetes tests - that make much more sense so that the other tests will not be affected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#discussion_r336760933 ## File path: .travis.yml ## @@ -63,8 +43,28 @@ jobs: - name: "Build documentation" stage: pre-test script: ./scripts/ci/ci_docs.sh +- name: "Tests postgres kubernetes python 3.6 (persistent)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=persistent_mode + python: "3.6" + stage: test +- name: "Tests postgres kubernetes python 3.6 (git)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=git_mode + python: "3.6" + stage: test +- name: "Tests postgres python 3.6" + env: BACKEND=postgres ENV=docker + python: "3.6" + stage: test +- name: "Tests sqlite python 3.5" + env: BACKEND=sqlite ENV=docker + python: "3.5" + stage: test +- name: "Tests mysql python 3.7" + env: BACKEND=mysql ENV=docker + python: "3.7" + stage: test services: - docker before_install: - ./scripts/ci/ci_before_install.sh -script: "./scripts/ci/ci_run_airflow_testing.sh" +script: travis_wait 30 "./scripts/ci/ci_run_airflow_testing.sh" Review comment: That's exactly the case. Right now in kubernetes tests we have more than 20 minutes to wait for the deployment without writing anything to output. This kills Travis builds. travis_wait is there to prolong that - side effect of travis_wait however is that it does not write ANYTHING to output until it is finished (which I found actually quite interesting - I rarely look at the output while it is running now. This will change in the future anyway if we switch to another build system. Now that we have kind working, I will be able to switch back to trying GitLab + Kubernetes (or maybe even Github Actions) and make it more robust and print some output while it is being built (especially after Tomek's pytest changes). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
potiuk commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#issuecomment-544215001 > Is this optimization based on reducing docker layers? Is this the reason for this change? See the responses above @mik-laj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
potiuk commented on a change in pull request #6373: Depends on [AIRFLOW-5680]
[AIRFLOW-5704] Optimise kubernetes image
URL: https://github.com/apache/airflow/pull/6373#discussion_r336760858
##
File path: Dockerfile
##
@@ -174,39 +170,8 @@ RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci"
]]; then \
;\
fi
-# TODO: We should think about removing those and moving them into
docker-compose dependencies.
COPY scripts/ci/docker_build/ci_build_install_deps.sh
/tmp/ci_build_install_deps.sh
-# Kubernetes dependencies
-RUN \
-if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
Review comment:
It's not at runtime - it's just grouping all the CI-related installs in
single RUN command.
This is mostly to decrease the build time for slim (in the future
production) image. The problem is that currently in multi-staging images, even
if you have a stage that is not used, it will not be skipped (this will be
improved when BUILDKIT goes out of beta and we will be able to use it including
caching - in BUILDKIT, unused stages are skipped while building). The "if"
there is a hack to skip building CI dependencies if we are only building slim
image.
Separating it to single script makes it slightly faster to rebuild the slim
image- every RUN command takes couple of seconds when executed so if we have a
separate RUN and if in every RUN, we got 20-30 seconds longer builds always
(because this stage is not part of the image, so we do not have cache for it,
it is also not cached so it is always built - even if we locally built the slim
image. Unfortunately, we cannot use both --cache-from and local cache at the
same time (https://github.com/moby/moby/issues/32612) - this could also solve
the problem.
This will also be useful in the PROD image change I am working on.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5694) Check for blinker when detecting Sentry packages
[
https://issues.apache.org/jira/browse/AIRFLOW-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955355#comment-16955355
]
ASF GitHub Bot commented on AIRFLOW-5694:
-
potiuk commented on pull request #6365: [AIRFLOW-5694] Check for blinker in
Sentry setup
URL: https://github.com/apache/airflow/pull/6365
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Check for blinker when detecting Sentry packages
>
>
> Key: AIRFLOW-5694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5694
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Minor
> Fix For: 1.10.6
>
>
> After upgrading to 1.10.6rc1 with `sentry-sdk` installed but not specifying
> the `[sentry]` extra, the dependency `blinker` will cause failures of the
> following form:
> {code:python}
> ../lib/python3.7/site-packages/airflow/__init__.py:40: in
> from airflow.models import DAG
> ../lib/python3.7/site-packages/airflow/models/__init__.py:21: in
> from airflow.models.baseoperator import BaseOperator # noqa: F401
> ../lib/python3.7/site-packages/airflow/models/baseoperator.py:42: in
> from airflow.models.dag import DAG
> ../lib/python3.7/site-packages/airflow/models/dag.py:51: in
> from airflow.models.taskinstance import TaskInstance, clear_task_instances
> ../lib/python3.7/site-packages/airflow/models/taskinstance.py:53: in
> from airflow.sentry import Sentry
> ../lib/python3.7/site-packages/airflow/sentry.py:167: in
> Sentry = ConfiguredSentry()
> ../lib/python3.7/site-packages/airflow/sentry.py:94: in __init__
> init(integrations=integrations)
> ../lib/python3.7/site-packages/sentry_sdk/hub.py:81: in _init
> client = Client(*args, **kwargs) # type: ignore
> ../lib/python3.7/site-packages/sentry_sdk/client.py:80: in __init__
> self._init_impl()
> ../lib/python3.7/site-packages/sentry_sdk/client.py:108: in _init_impl
> with_defaults=self.options["default_integrations"],
> ../lib/python3.7/site-packages/sentry_sdk/integrations/__init__.py:82: in
> setup_integrations
> type(integration).setup_once()
> ../lib/python3.7/site-packages/sentry_sdk/integrations/flask.py:57: in
> setup_once
> appcontext_pushed.connect(_push_appctx)
> ../lib/python3.7/site-packages/flask/signals.py:39: in _fail
> "Signalling support is unavailable because the blinker"
> E RuntimeError: Signalling support is unavailable because the blinker
> library is not installed.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5694) Check for blinker when detecting Sentry packages
[
https://issues.apache.org/jira/browse/AIRFLOW-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955356#comment-16955356
]
ASF subversion and git services commented on AIRFLOW-5694:
--
Commit c72c42730236fee1526fcc03dca7f88e1778ee94 in airflow's branch
refs/heads/master from Marcus Levine
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c72c427 ]
[AIRFLOW-5694] Check for blinker in Sentry setup (#6365)
> Check for blinker when detecting Sentry packages
>
>
> Key: AIRFLOW-5694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5694
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcus Levine
>Assignee: Marcus Levine
>Priority: Minor
> Fix For: 1.10.6
>
>
> After upgrading to 1.10.6rc1 with `sentry-sdk` installed but not specifying
> the `[sentry]` extra, the dependency `blinker` will cause failures of the
> following form:
> {code:python}
> ../lib/python3.7/site-packages/airflow/__init__.py:40: in
> from airflow.models import DAG
> ../lib/python3.7/site-packages/airflow/models/__init__.py:21: in
> from airflow.models.baseoperator import BaseOperator # noqa: F401
> ../lib/python3.7/site-packages/airflow/models/baseoperator.py:42: in
> from airflow.models.dag import DAG
> ../lib/python3.7/site-packages/airflow/models/dag.py:51: in
> from airflow.models.taskinstance import TaskInstance, clear_task_instances
> ../lib/python3.7/site-packages/airflow/models/taskinstance.py:53: in
> from airflow.sentry import Sentry
> ../lib/python3.7/site-packages/airflow/sentry.py:167: in
> Sentry = ConfiguredSentry()
> ../lib/python3.7/site-packages/airflow/sentry.py:94: in __init__
> init(integrations=integrations)
> ../lib/python3.7/site-packages/sentry_sdk/hub.py:81: in _init
> client = Client(*args, **kwargs) # type: ignore
> ../lib/python3.7/site-packages/sentry_sdk/client.py:80: in __init__
> self._init_impl()
> ../lib/python3.7/site-packages/sentry_sdk/client.py:108: in _init_impl
> with_defaults=self.options["default_integrations"],
> ../lib/python3.7/site-packages/sentry_sdk/integrations/__init__.py:82: in
> setup_integrations
> type(integration).setup_once()
> ../lib/python3.7/site-packages/sentry_sdk/integrations/flask.py:57: in
> setup_once
> appcontext_pushed.connect(_push_appctx)
> ../lib/python3.7/site-packages/flask/signals.py:39: in _fail
> "Signalling support is unavailable because the blinker"
> E RuntimeError: Signalling support is unavailable because the blinker
> library is not installed.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6365: [AIRFLOW-5694] Check for blinker in Sentry setup
potiuk merged pull request #6365: [AIRFLOW-5694] Check for blinker in Sentry setup URL: https://github.com/apache/airflow/pull/6365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5680. --- Fix Version/s: 2.0.0 Resolution: Fixed > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5706) Add KubernetesPodYamlOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955333#comment-16955333
]
ASF GitHub Bot commented on AIRFLOW-5706:
-
mik-laj commented on pull request #6375: [AIRFLOW-5706] Add
KubernetesPodYamlOperator
URL: https://github.com/apache/airflow/pull/6375
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add KubernetesPodYamlOperator
> -
>
> Key: AIRFLOW-5706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5706
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
> I would like to create an operator that allows us to run pods using the yaml
> file definition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6375: [AIRFLOW-5706] Add KubernetesPodYamlOperator
mik-laj opened a new pull request #6375: [AIRFLOW-5706] Add
KubernetesPodYamlOperator
URL: https://github.com/apache/airflow/pull/6375
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5706) Add KubernetesPodYamlOperator
Kamil Bregula created AIRFLOW-5706: -- Summary: Add KubernetesPodYamlOperator Key: AIRFLOW-5706 URL: https://issues.apache.org/jira/browse/AIRFLOW-5706 Project: Apache Airflow Issue Type: New Feature Components: operators Affects Versions: 1.10.5 Reporter: Kamil Bregula I would like to create an operator that allows us to run pods using the yaml file definition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy
codecov-io commented on issue #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy URL: https://github.com/apache/airflow/pull/6374#issuecomment-544203724 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=h1) Report > Merging [#6374](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6374 +/- ## == + Coverage 80.39% 80.41% +0.01% == Files 616 616 Lines 3580435804 == + Hits2878328790 +7 + Misses 7021 7014 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/app.py](https://codecov.io/gh/apache/airflow/pull/6374/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5) | `96.49% <100%> (-0.07%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6374/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.22% <100%> (+0.03%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6374/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.79% <0%> (+0.5%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6374/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=footer). Last update [c19a60c...011ad08](https://codecov.io/gh/apache/airflow/pull/6374?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
mik-laj commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#discussion_r336756383 ## File path: .travis.yml ## @@ -63,8 +43,28 @@ jobs: - name: "Build documentation" stage: pre-test script: ./scripts/ci/ci_docs.sh +- name: "Tests postgres kubernetes python 3.6 (persistent)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=persistent_mode + python: "3.6" + stage: test +- name: "Tests postgres kubernetes python 3.6 (git)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=git_mode + python: "3.6" + stage: test +- name: "Tests postgres python 3.6" + env: BACKEND=postgres ENV=docker + python: "3.6" + stage: test +- name: "Tests sqlite python 3.5" + env: BACKEND=sqlite ENV=docker + python: "3.5" + stage: test +- name: "Tests mysql python 3.7" + env: BACKEND=mysql ENV=docker + python: "3.7" + stage: test services: - docker before_install: - ./scripts/ci/ci_before_install.sh -script: "./scripts/ci/ci_run_airflow_testing.sh" +script: travis_wait 30 "./scripts/ci/ci_run_airflow_testing.sh" Review comment: Something goes on in the background, but I don't write anything on the screen. In this case, Travis stops the job. This program causes a dot to be printed on the screen from time to time, which means the program has more time to execute. I don't know the details of this situation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
mik-laj commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-544203312 I restarted failed job. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
mik-laj commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-544203299 It looks like some kind of dependencies has exploded again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5705) add option for alternative creds backend
Daniel Standish created AIRFLOW-5705: Summary: add option for alternative creds backend Key: AIRFLOW-5705 URL: https://issues.apache.org/jira/browse/AIRFLOW-5705 Project: Apache Airflow Issue Type: New Feature Components: core Affects Versions: 1.10.5 Reporter: Daniel Standish Assignee: Daniel Standish Idea hear is to create some kind of generic creds backend that could support using other creds stores such as AWS SSM parameter store. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues
codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544200778 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=h1) Report > Merging [#6372](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6372 +/- ## == + Coverage 80.39% 80.41% +0.01% == Files 616 616 Lines 3580435804 == + Hits2878328790 +7 + Misses 7021 7014 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <ø> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.62% <ø> (ø)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `75.6% <ø> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.21% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YXByb2MucHk=) | `55.25% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `82.17% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_firehose\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19maXJlaG9zZV9ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `91.93% <ø> (ø)` | :arrow_up: | | ... and [35 more](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=footer). Last update [c19a60c...d781107](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues
codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544200778 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=h1) Report > Merging [#6372](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6372 +/- ## == + Coverage 80.39% 80.41% +0.01% == Files 616 616 Lines 3580435804 == + Hits2878328790 +7 + Misses 7021 7014 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <ø> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.62% <ø> (ø)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `75.6% <ø> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.21% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YXByb2MucHk=) | `55.25% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `82.17% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_firehose\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19maXJlaG9zZV9ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `91.93% <ø> (ø)` | :arrow_up: | | ... and [35 more](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=footer). Last update [c19a60c...d781107](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues
codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544200778 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=h1) Report > Merging [#6372](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6372 +/- ## == + Coverage 80.39% 80.41% +0.01% == Files 616 616 Lines 3580435804 == + Hits2878328790 +7 + Misses 7021 7014 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <ø> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.62% <ø> (ø)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `75.6% <ø> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.21% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YXByb2MucHk=) | `55.25% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `82.17% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_firehose\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19maXJlaG9zZV9ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `91.93% <ø> (ø)` | :arrow_up: | | ... and [35 more](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=footer). Last update [c19a60c...d781107](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues
codecov-io edited a comment on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544200778 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=h1) Report > Merging [#6372](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6372 +/- ## == + Coverage 80.39% 80.41% +0.01% == Files 616 616 Lines 3580435804 == + Hits2878328790 +7 + Misses 7021 7014 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <ø> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.62% <ø> (ø)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `75.6% <ø> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.21% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YXByb2MucHk=) | `55.25% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `82.17% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_firehose\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19maXJlaG9zZV9ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `91.93% <ø> (ø)` | :arrow_up: | | ... and [35 more](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=footer). Last update [c19a60c...d781107](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6372: [AIRFLOW-5702] Fix common docstring issues
codecov-io commented on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544200778 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=h1) Report > Merging [#6372](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c19a60c23df9dd4fab42e68e9a1992f7f89c281c?src=pr=desc) will **decrease** coverage by `0.24%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6372 +/- ## == - Coverage 80.39% 80.14% -0.25% == Files 616 616 Lines 3580435804 == - Hits2878328696 -87 - Misses 7021 7108 +87 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <ø> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.62% <ø> (ø)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `75.6% <ø> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.21% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YXByb2MucHk=) | `55.25% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `82.17% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/aws\_firehose\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19maXJlaG9zZV9ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `91.93% <ø> (ø)` | :arrow_up: | | ... and [41 more](https://codecov.io/gh/apache/airflow/pull/6372/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=footer). Last update [c19a60c...d781107](https://codecov.io/gh/apache/airflow/pull/6372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4470) RBAC Github Enterprise OAuth provider callback URL?
[
https://issues.apache.org/jira/browse/AIRFLOW-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955297#comment-16955297
]
Vince commented on AIRFLOW-4470:
I wasn't able to find this in the documentation anywhere, either, but the
answer seems to be
{noformat}
https://your-airflow-url/oauth-authorized/github{noformat}
> RBAC Github Enterprise OAuth provider callback URL?
> ---
>
> Key: AIRFLOW-4470
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4470
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, webserver
>Affects Versions: 1.10.2
>Reporter: Geez
>Priority: Blocker
> Labels: usability
>
> Hi all,
> Quick question, when using RBAC with OAuth providers (1.10.2):
> * we are not specifying the {{authenticate}} or {{auth_backend}} in the
> [webserver] section of \{{airflow.cfg}}anymore
> * Instead, we set the OAuth provider config in the flask-appbuilder's
> {{webserver_config.py}}:
> {code:java}
>
> # Adapting Google OAuth example to Github:
> OAUTH_PROVIDERS = [
> {'name':'github', 'icon':'fa-github', 'token_key':'access_token',
> 'remote_app': {
> 'base_url':'https://github.corporate-domain.com/login',
>
> 'access_token_url':'https://github.corporate-domain.com/login/oauth/access_token',
>
> 'authorize_url':'https://github.corporate-domain.com/login/oauth/authorize',
> 'request_token_url': None,
> 'consumer_key': '',
> 'consumer_secret': 'X',
> }
> }
> ]
>
> {code}
> _Question:_
> * so what callback URL do we specify in the app?
> {{http:/webapp/ghe_oauth/callback}} would not work right? (example with
> github entreprise)
> No matter what I specify for the callback url (/ghe_oauth/callback or
> [http://webapp.com|http://webapp.com/]), I get an error message about
> {{redirect_uri}} mismatch:
> {code:java}
> {{error=redirect_uri_mismatch_description=The+redirect_uri+MUST+match+the+registered+callback+URL+for+this+application
> }}{code}
> _Docs ref:_
> Here is how you setup OAuth with Github Entreprise on Airflow _*without*_
> RBAC:
> [https://airflow.apache.org/security.html#github-enterprise-ghe-authentication]
> And here is how you setup OAuth via the {{webserver_config.py}} of
> flask_appbuilder used by airflow _*with*_RBAC:
>
> [https://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-oauth]
> What's the *callback url* when using RBAC and OAuth with Airflow?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] osule commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy
osule commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken
links when running behind reverse proxy
URL: https://github.com/apache/airflow/pull/6374#discussion_r336753742
##
File path: airflow/www/static/css/bootstrap-theme.css
##
@@ -3252,7 +3252,8 @@ tbody.collapse.in {
content: "\e223";
}
.glyphicon-duplicate:before {
- content: "\e224";
+ font-family: "FontAwesome";
Review comment:
Sure, I wasn't aware this was taken care of already in a different PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
dimberman commented on a change in pull request #6373: Depends on
[AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
URL: https://github.com/apache/airflow/pull/6373#discussion_r336753027
##
File path: Dockerfile
##
@@ -174,39 +170,8 @@ RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci"
]]; then \
;\
fi
-# TODO: We should think about removing those and moving them into
docker-compose dependencies.
COPY scripts/ci/docker_build/ci_build_install_deps.sh
/tmp/ci_build_install_deps.sh
-# Kubernetes dependencies
-RUN \
-if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
Review comment:
So is the idea here to remove these from the CI image and place them in the
build script so they are loaded at run-time? I'm not sure what we gain from
that?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6365: [AIRFLOW-5694] Check for blinker in Sentry setup
codecov-io edited a comment on issue #6365: [AIRFLOW-5694] Check for blinker in Sentry setup URL: https://github.com/apache/airflow/pull/6365#issuecomment-544195447 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=h1) Report > Merging [#6365](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6365 +/- ## = + Coverage 80.09% 80.1% +<.01% = Files 616 616 Lines 35804 35805 +1 = + Hits28679 28680 +1 Misses 71257125 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/6365/diff?src=pr=tree#diff-YWlyZmxvdy9zZW50cnkucHk=) | `89.04% <100%> (+0.15%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=footer). Last update [b8c0263...7fd39cf](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6365: [AIRFLOW-5694] Check for blinker in Sentry setup
codecov-io commented on issue #6365: [AIRFLOW-5694] Check for blinker in Sentry setup URL: https://github.com/apache/airflow/pull/6365#issuecomment-544195447 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=h1) Report > Merging [#6365](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6365 +/- ## = + Coverage 80.09% 80.1% +<.01% = Files 616 616 Lines 35804 35805 +1 = + Hits28679 28680 +1 Misses 71257125 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/6365/diff?src=pr=tree#diff-YWlyZmxvdy9zZW50cnkucHk=) | `89.04% <100%> (+0.15%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=footer). Last update [b8c0263...7fd39cf](https://codecov.io/gh/apache/airflow/pull/6365?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
dimberman commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#discussion_r336752961 ## File path: .travis.yml ## @@ -63,8 +43,28 @@ jobs: - name: "Build documentation" stage: pre-test script: ./scripts/ci/ci_docs.sh +- name: "Tests postgres kubernetes python 3.6 (persistent)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=persistent_mode + python: "3.6" + stage: test +- name: "Tests postgres kubernetes python 3.6 (git)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=git_mode + python: "3.6" + stage: test +- name: "Tests postgres python 3.6" + env: BACKEND=postgres ENV=docker + python: "3.6" + stage: test +- name: "Tests sqlite python 3.5" + env: BACKEND=sqlite ENV=docker + python: "3.5" + stage: test +- name: "Tests mysql python 3.7" + env: BACKEND=mysql ENV=docker + python: "3.7" + stage: test services: - docker before_install: - ./scripts/ci/ci_before_install.sh -script: "./scripts/ci/ci_run_airflow_testing.sh" +script: travis_wait 30 "./scripts/ci/ci_run_airflow_testing.sh" Review comment: Wait I'm confused, why is there a 30 minute wait here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
nuclearpinguin commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-544193463 @mik-laj, @potiuk can you take a look at the Travis? I am not sure if this is something flaky or not. https://api.travis-ci.org/v3/job/53904/log.txt This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
codecov-io edited a comment on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#issuecomment-544193294 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=h1) Report > Merging [#6373](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **decrease** coverage by `70.58%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) ```diff @@Coverage Diff @@ ## master #6373 +/- ## == - Coverage 80.09% 9.51% -70.59% == Files 616 616 Lines 35804 35804 == - Hits286793407-25272 - Misses 7125 32397+25272 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/k8s\_model.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2s4c19tb2RlbC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/sensors/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avc2Vuc29ycy9iaWdxdWVyeV9kdHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/operators/text\_to\_speech.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL3RleHRfdG9fc3BlZWNoLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/hooks/discovery\_api.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGlzY292ZXJ5X2FwaS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/file\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9maWxlX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | ... and [508 more](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=footer). Last update [b8c0263...8fd37ed](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
codecov-io commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#issuecomment-544193294 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=h1) Report > Merging [#6373](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **decrease** coverage by `70.58%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) ```diff @@Coverage Diff @@ ## master #6373 +/- ## == - Coverage 80.09% 9.51% -70.59% == Files 616 616 Lines 35804 35804 == - Hits286793407-25272 - Misses 7125 32397+25272 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/k8s\_model.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2s4c19tb2RlbC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/sensors/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avc2Vuc29ycy9iaWdxdWVyeV9kdHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/operators/text\_to\_speech.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL3RleHRfdG9fc3BlZWNoLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/gcp/hooks/discovery\_api.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGlzY292ZXJ5X2FwaS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/file\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9maWxlX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | ... and [508 more](https://codecov.io/gh/apache/airflow/pull/6373/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=footer). Last update [b8c0263...8fd37ed](https://codecov.io/gh/apache/airflow/pull/6373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
mik-laj commented on a change in pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#discussion_r336751552 ## File path: .travis.yml ## @@ -63,8 +43,28 @@ jobs: - name: "Build documentation" stage: pre-test script: ./scripts/ci/ci_docs.sh +- name: "Tests postgres kubernetes python 3.6 (persistent)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=persistent_mode + python: "3.6" + stage: test +- name: "Tests postgres kubernetes python 3.6 (git)" + env: BACKEND=postgres ENV=kubernetes KUBERNETES_MODE=git_mode + python: "3.6" + stage: test +- name: "Tests postgres python 3.6" + env: BACKEND=postgres ENV=docker + python: "3.6" + stage: test +- name: "Tests sqlite python 3.5" + env: BACKEND=sqlite ENV=docker + python: "3.5" + stage: test +- name: "Tests mysql python 3.7" + env: BACKEND=mysql ENV=docker + python: "3.7" + stage: test services: - docker before_install: - ./scripts/ci/ci_before_install.sh -script: "./scripts/ci/ci_run_airflow_testing.sh" +script: travis_wait 30 "./scripts/ci/ci_run_airflow_testing.sh" Review comment: I'm glad you found this parameter that I told you about. The number 30 increases the waiting time to 30 minutes compared to 20 minutes by default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955261#comment-16955261 ] ASF subversion and git services commented on AIRFLOW-5680: -- Commit c19a60c23df9dd4fab42e68e9a1992f7f89c281c in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c19a60c ] [AIRFLOW-5680] Fixes Kubernetes hangs (#6347) > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[ https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955260#comment-16955260 ] ASF GitHub Bot commented on AIRFLOW-5680: - mik-laj commented on pull request #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kubernetes builds are often timing out even with kind > - > > Key: AIRFLOW-5680 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5680 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6347: [AIRFLOW-5680] Fixes Kubernetes hangs
mik-laj merged pull request #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs
mik-laj commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347#issuecomment-544191148 I am afraid that it will be difficult to test these changes on a separate branch. It does not look like it could make things worse, but there is a chance that the problems with Kubernetes will be reduced, so I am merging these changes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
mik-laj edited a comment on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#issuecomment-544190511 Is this optimization based on reducing docker layers? Is this the reason for this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs
codecov-io commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347#issuecomment-544190567 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=h1) Report > Merging [#6347](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **increase** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6347 +/- ## + Coverage 80.09% 80.4% +0.3% Files 616 616 Lines 35804 35804 + Hits28679 28787+108 + Misses 71257017-108 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/k8s\_model.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2s4c19tb2RlbC5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.4% <0%> (-0.5%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.5% <0%> (+28.35%)` | :arrow_up: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `75% <0%> (+41.66%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `91.97% <0%> (+46.71%)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `100% <0%> (+47.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6347/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `100% <0%> (+55.55%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=footer). Last update [b8c0263...7586dee](https://codecov.io/gh/apache/airflow/pull/6347?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
mik-laj commented on issue #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image URL: https://github.com/apache/airflow/pull/6373#issuecomment-544190511 Does this change reduce the number of Docker layers, resulting in time optimization? Is this the reason for this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy
mik-laj commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken
links when running behind reverse proxy
URL: https://github.com/apache/airflow/pull/6374#discussion_r336750528
##
File path: airflow/www/static/css/bootstrap-theme.css
##
@@ -3252,7 +3252,8 @@ tbody.collapse.in {
content: "\e223";
}
.glyphicon-duplicate:before {
- content: "\e224";
+ font-family: "FontAwesome";
Review comment:
This is an unrelated change. Can you revert it? We are working on full
font-awesome migration: https://github.com/apache/airflow/pull/6254
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy
mik-laj commented on a change in pull request #6374: [AIRFLOW-5658] Fix broken
links when running behind reverse proxy
URL: https://github.com/apache/airflow/pull/6374#discussion_r336750528
##
File path: airflow/www/static/css/bootstrap-theme.css
##
@@ -3252,7 +3252,8 @@ tbody.collapse.in {
content: "\e223";
}
.glyphicon-duplicate:before {
- content: "\e224";
+ font-family: "FontAwesome";
Review comment:
This is an unrelated change. Can you revert it? We are working on full
migration to font-awesome: https://github.com/apache/airflow/pull/6254
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers.
codecov-io edited a comment on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers. URL: https://github.com/apache/airflow/pull/6196#issuecomment-536021136 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=h1) Report > Merging [#6196](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **decrease** coverage by `0.14%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6196 +/- ## == - Coverage 80.09% 79.95% -0.15% == Files 616 616 Lines 3580435805 +1 == - Hits2867928627 -52 - Misses 7125 7178 +53 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.58% <100%> (+0.43%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `72.72% <0%> (-20.46%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.26% <0%> (-4.09%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.23% <0%> (-2.67%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `73.23% <0%> (-1.21%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=footer). Last update [b8c0263...0905fa6](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers.
codecov-io edited a comment on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers. URL: https://github.com/apache/airflow/pull/6196#issuecomment-536021136 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=h1) Report > Merging [#6196](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b8c02632136320b8379956411134246cd2f6eb47?src=pr=desc) will **decrease** coverage by `0.14%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6196 +/- ## == - Coverage 80.09% 79.95% -0.15% == Files 616 616 Lines 3580435805 +1 == - Hits2867928627 -52 - Misses 7125 7178 +53 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.58% <100%> (+0.43%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `72.72% <0%> (-20.46%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.26% <0%> (-4.09%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.23% <0%> (-2.67%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `73.23% <0%> (-1.21%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6196/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=footer). Last update [b8c0263...0905fa6](https://codecov.io/gh/apache/airflow/pull/6196?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] osule opened a new pull request #6374: [AIRFLOW-5658] Fix broken links when running behind reverse proxy
osule opened a new pull request #6374: [AIRFLOW-5658] Fix broken links when
running behind reverse proxy
URL: https://github.com/apache/airflow/pull/6374
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/5658) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-5658
- ~In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.~
- ~In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).~
- ~In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).~
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This fixes broken links to Configurations and Versions page when running
Airflow behind reverse proxy.
Also, broken display of Task tries icon is also remedied.
__Before__
https://user-images.githubusercontent.com/13223818/67150097-a272a180-f2b3-11e9-83b6-a8f5a3000b8c.png;>
__After__
https://user-images.githubusercontent.com/13223818/67150078-62abba00-f2b3-11e9-9955-8c5bacf008a7.png;>
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: No fundamental changes to code.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] ~In case of new functionality, my PR adds documentation that describes
how to use it.~
- ~All the public functions and the classes in the PR contain docstrings
that explain what it does~
- ~If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5658) Links issue through reverse proxy access
[
https://issues.apache.org/jira/browse/AIRFLOW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955257#comment-16955257
]
ASF GitHub Bot commented on AIRFLOW-5658:
-
osule commented on pull request #6374: [AIRFLOW-5658] Fix broken links when
running behind reverse proxy
URL: https://github.com/apache/airflow/pull/6374
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/5658) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-5658
- ~In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.~
- ~In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).~
- ~In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).~
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This fixes broken links to Configurations and Versions page when running
Airflow behind reverse proxy.
Also, broken display of Task tries icon is also remedied.
__Before__
https://user-images.githubusercontent.com/13223818/67150097-a272a180-f2b3-11e9-83b6-a8f5a3000b8c.png;>
__After__
https://user-images.githubusercontent.com/13223818/67150078-62abba00-f2b3-11e9-9955-8c5bacf008a7.png;>
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: No fundamental changes to code.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] ~In case of new functionality, my PR adds documentation that describes
how to use it.~
- ~All the public functions and the classes in the PR contain docstrings
that explain what it does~
- ~If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Links issue through reverse proxy access
> -
>
> Key: AIRFLOW-5658
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5658
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4, 1.10.5
>Reporter: Mikołaj Morawski
>Assignee: Oluwafemi Sule
>Priority: Trivial
> Attachments: image-2019-10-15-16-02-45-731.png
>
>
> The access through a reverse proxy is not working for the following two uris:
> /configuration
> /version
>
> I think that the root cause is here:
> [https://github.com/apache/airflow/blob/master/airflow/www/app.py]
> {code:java}
> appbuilder.add_link("Configurations",
> href='/configuration',
> category="Admin",
> category_icon="fa-user")
> appbuilder.add_link('Version',
> href='/version',
> category='About',
> category_icon='fa-th')
> {code}
> the "href" parameter does not use the "views" reference here. The add_link
> should not be used for this reference.
>
> The second problem is that the "Task Tries" icon is not displayed properly.
> !image-2019-10-15-16-02-45-731.png!
> Regards,
> Mikolaj
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] iroddis commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling
iroddis commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling URL: https://github.com/apache/airflow/pull/5787#issuecomment-544184899 I haven't run the tests or tried to backport the functionality on older releases. All of the tests are currently passing, but I'm sure there are edge cases. I'd love to see this in a release, maybe tagged as an alpha feature, to get some wider testing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-5658) Links issue through reverse proxy access
[
https://issues.apache.org/jira/browse/AIRFLOW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oluwafemi Sule reassigned AIRFLOW-5658:
---
Assignee: Oluwafemi Sule
> Links issue through reverse proxy access
> -
>
> Key: AIRFLOW-5658
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5658
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4, 1.10.5
>Reporter: Mikołaj Morawski
>Assignee: Oluwafemi Sule
>Priority: Trivial
> Attachments: image-2019-10-15-16-02-45-731.png
>
>
> The access through a reverse proxy is not working for the following two uris:
> /configuration
> /version
>
> I think that the root cause is here:
> [https://github.com/apache/airflow/blob/master/airflow/www/app.py]
> {code:java}
> appbuilder.add_link("Configurations",
> href='/configuration',
> category="Admin",
> category_icon="fa-user")
> appbuilder.add_link('Version',
> href='/version',
> category='About',
> category_icon='fa-th')
> {code}
> the "href" parameter does not use the "views" reference here. The add_link
> should not be used for this reference.
>
> The second problem is that the "Task Tries" icon is not displayed properly.
> !image-2019-10-15-16-02-45-731.png!
> Regards,
> Mikolaj
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5704) Docker image for kind kubernetes tests can be optimised
Jarek Potiuk created AIRFLOW-5704: - Summary: Docker image for kind kubernetes tests can be optimised Key: AIRFLOW-5704 URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk The docker CI image for kind tests can be optimised a bit - Kubernetes Version should be removed and all the installation of docker + kubectl + kind can be moved to ci_install* script -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5680) Kubernetes builds are often timing out even with kind
[
https://issues.apache.org/jira/browse/AIRFLOW-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955239#comment-16955239
]
ASF GitHub Bot commented on AIRFLOW-5680:
-
potiuk commented on pull request #6373: Depends on [AIRFLOW-5680]
[AIRFLOW-5704] Optimise kubernetes image
URL: https://github.com/apache/airflow/pull/6373
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kubernetes builds are often timing out even with kind
> -
>
> Key: AIRFLOW-5680
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5680
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0, 1.10.5
>Reporter: Jarek Potiuk
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6373: Depends on [AIRFLOW-5680] [AIRFLOW-5704] Optimise kubernetes image
potiuk opened a new pull request #6373: Depends on [AIRFLOW-5680]
[AIRFLOW-5704] Optimise kubernetes image
URL: https://github.com/apache/airflow/pull/6373
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs
potiuk commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347#issuecomment-544178454 @gerardo -> seems that it was just even longer timeout to set :). I hope it will all be ok now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs
potiuk commented on issue #6347: [AIRFLOW-5680] Fixes Kubernetes hangs URL: https://github.com/apache/airflow/pull/6347#issuecomment-544177747 This should fix the Kubernetes problem :) Now Kind builds should work just fine. We need to optimise them later as they are now taking 33 minutes (but I already know how I think) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
nuclearpinguin commented on a change in pull request #6371: [AIRFLOW-5691]
Rewrite Dataproc operators to use python library
URL: https://github.com/apache/airflow/pull/6371#discussion_r336746250
##
File path: airflow/gcp/example_dags/example_dataproc.py
##
@@ -41,69 +40,130 @@
PYSPARK_MAIN = os.environ.get("PYSPARK_MAIN", "hello_world.py")
PYSPARK_URI = "gs://{}/{}".format(BUCKET, PYSPARK_MAIN)
+
+# Cluster definition
+CLUSTER = {
+"project_id": PROJECT_ID,
+"cluster_name": CLUSTER_NAME,
+"config": {
+"master_config": {
+"num_instances": 1,
+"machine_type_uri": "n1-standard-4",
+"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 1024},
+},
+"worker_config": {
+"num_instances": 2,
+"machine_type_uri": "n1-standard-4",
+"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 1024},
+},
+},
+}
+
+
+# Update options
+CLUSTER_UPDATE = {
+"config": {
+"worker_config": {"num_instances": 3},
+"secondary_worker_config": {"num_instances": 3},
+}
+}
+UPDATE_MASK = {
+"paths": [
+"config.worker_config.num_instances",
+"config.secondary_worker_config.num_instances",
+]
+}
+
+TIMEOUT = {"seconds": 1 * 24 * 60 * 60}
+
+
+# Jobs definitions
+PIG_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"pig_job": {"query_list": {"queries": ["define sin HiveUDF('sin');"]}},
+}
+
+SPARK_SQL_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"spark_sql_job": {"query_list": {"queries": ["SHOW DATABASES;"]}},
+}
+
+SPARK_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"spark_job": {
+"jar_file_uris":
["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
+"main_class": "org.apache.spark.examples.SparkPi",
+},
+}
+
+PYSPARK_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"pyspark_job": {"main_python_file_uri": PYSPARK_URI},
+}
+
+HIVE_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"hive_job": {"query_list": {"queries": ["SHOW DATABASES;"]}},
+}
+
+HADOOP_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"hadoop_job": {
+"main_jar_file_uri":
"file:///usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar",
+"args": ["wordcount", "gs://pub/shakespeare/rose.txt", OUTPUT_PATH],
+},
+}
+
with models.DAG(
"example_gcp_dataproc",
default_args={"start_date": airflow.utils.dates.days_ago(1)},
schedule_interval=None,
) as dag:
create_cluster = DataprocClusterCreateOperator(
-task_id="create_cluster",
-cluster_name=CLUSTER_NAME,
-project_id=PROJECT_ID,
-num_workers=2,
-region=REGION,
+task_id="create_cluster", project_id=PROJECT_ID, cluster=CLUSTER,
region=REGION
Review comment:
Yes, that was how I checked backwards compatibility. Channing example DAG
was last thing I did. If you wish I can change the DAG in separate PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
mik-laj commented on a change in pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#discussion_r336746007 ## File path: airflow/gcp/hooks/dataproc.py ## @@ -498,215 +836,59 @@ def submit( :type region: str :param job_error_states: Job states that should be considered error states. :type job_error_states: List[str] -:raises: Excepion -""" -submitted = _DataProcJob(self.get_conn(), project_id, job, region, - job_error_states=job_error_states, - num_retries=self.num_retries) -if not submitted.wait_for_done(): -submitted.raise_error() - -def create_job_template( -self, -task_id: str, -cluster_name: str, -job_type: str, -properties: Dict[str, str], -) -> _DataProcJobBuilder: """ -Creates Google Cloud Dataproc job template. - -:param task_id: id of the task -:type task_id: str -:param cluster_name: Dataproc cluster name. -:type cluster_name: str -:param job_type: Type of Dataproc job. -:type job_type: str -:param properties: Additional properties of the job. -:type properties: dict -:return: Dataproc Job -""" -if not self.project_id: -raise AirflowException( -"The project ID could not be determined. You should specify the project id " -"in the connection configuration." -) -return _DataProcJobBuilder( -self.project_id, -task_id, -cluster_name, -job_type, -properties +# TODO: Remover one day +warnings.warn( +"This method is deprecated. Please use `submit_job`", +DeprecationWarning, Review comment: ```suggestion DeprecationWarning, stacklevel=2 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library
mik-laj commented on a change in pull request #6371: [AIRFLOW-5691] Rewrite
Dataproc operators to use python library
URL: https://github.com/apache/airflow/pull/6371#discussion_r336745852
##
File path: airflow/gcp/example_dags/example_dataproc.py
##
@@ -41,69 +40,130 @@
PYSPARK_MAIN = os.environ.get("PYSPARK_MAIN", "hello_world.py")
PYSPARK_URI = "gs://{}/{}".format(BUCKET, PYSPARK_MAIN)
+
+# Cluster definition
+CLUSTER = {
+"project_id": PROJECT_ID,
+"cluster_name": CLUSTER_NAME,
+"config": {
+"master_config": {
+"num_instances": 1,
+"machine_type_uri": "n1-standard-4",
+"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 1024},
+},
+"worker_config": {
+"num_instances": 2,
+"machine_type_uri": "n1-standard-4",
+"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 1024},
+},
+},
+}
+
+
+# Update options
+CLUSTER_UPDATE = {
+"config": {
+"worker_config": {"num_instances": 3},
+"secondary_worker_config": {"num_instances": 3},
+}
+}
+UPDATE_MASK = {
+"paths": [
+"config.worker_config.num_instances",
+"config.secondary_worker_config.num_instances",
+]
+}
+
+TIMEOUT = {"seconds": 1 * 24 * 60 * 60}
+
+
+# Jobs definitions
+PIG_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"pig_job": {"query_list": {"queries": ["define sin HiveUDF('sin');"]}},
+}
+
+SPARK_SQL_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"spark_sql_job": {"query_list": {"queries": ["SHOW DATABASES;"]}},
+}
+
+SPARK_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"spark_job": {
+"jar_file_uris":
["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
+"main_class": "org.apache.spark.examples.SparkPi",
+},
+}
+
+PYSPARK_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"pyspark_job": {"main_python_file_uri": PYSPARK_URI},
+}
+
+HIVE_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"hive_job": {"query_list": {"queries": ["SHOW DATABASES;"]}},
+}
+
+HADOOP_JOB = {
+"reference": {"project_id": PROJECT_ID},
+"placement": {"cluster_name": CLUSTER_NAME},
+"hadoop_job": {
+"main_jar_file_uri":
"file:///usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar",
+"args": ["wordcount", "gs://pub/shakespeare/rose.txt", OUTPUT_PATH],
+},
+}
+
with models.DAG(
"example_gcp_dataproc",
default_args={"start_date": airflow.utils.dates.days_ago(1)},
schedule_interval=None,
) as dag:
create_cluster = DataprocClusterCreateOperator(
-task_id="create_cluster",
-cluster_name=CLUSTER_NAME,
-project_id=PROJECT_ID,
-num_workers=2,
-region=REGION,
+task_id="create_cluster", project_id=PROJECT_ID, cluster=CLUSTER,
region=REGION
Review comment:
Have you tested this change with the original DAG?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages
feluelle commented on a change in pull request #6364: [AIRFLOW-5693] Support the "blocks" component for the Slack messages URL: https://github.com/apache/airflow/pull/6364#discussion_r336742705 ## File path: airflow/contrib/hooks/slack_webhook_hook.py ## @@ -42,6 +42,9 @@ class SlackWebhookHook(HttpHook): :param attachments: The attachments to send on Slack. Should be a list of dictionaries representing Slack attachments. :type attachments: list +:param blocks: The blocks to send on Slack. Should be a list of + dictionaries representing Slack blocks. Review comment: new lines there need to be indented with 4 Spaces. ```suggestion dictionaries representing Slack blocks. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #6372: [AIRFLOW-5702] Fix common docstring issues
feluelle commented on issue #6372: [AIRFLOW-5702] Fix common docstring issues URL: https://github.com/apache/airflow/pull/6372#issuecomment-544163266 A minor issue: `/opt/airflow/docs/_api/airflow/hooks/base_hook/index.rst:56: WARNING: Field list ends without a blank line; unexpected unindent.` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-5703) Airflow - SFTP Operator for multiple files
[ https://issues.apache.org/jira/browse/AIRFLOW-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5703 started by Mattia. --- > Airflow - SFTP Operator for multiple files > -- > > Key: AIRFLOW-5703 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5703 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.5 >Reporter: Mattia >Assignee: Mattia >Priority: Major > > *AS* User > *I WANT TO* download / upload multiple files from sftp server > *SO THAT* i need the possibility to add a list of file instead of single one -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5703) Airflow - SFTP Operator for multiple files
Mattia created AIRFLOW-5703: --- Summary: Airflow - SFTP Operator for multiple files Key: AIRFLOW-5703 URL: https://issues.apache.org/jira/browse/AIRFLOW-5703 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.5 Reporter: Mattia Assignee: Mattia *AS* User *I WANT TO* download / upload multiple files from sftp server *SO THAT* i need the possibility to add a list of file instead of single one -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5702) Fix common docstring issues
[
https://issues.apache.org/jira/browse/AIRFLOW-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955168#comment-16955168
]
ASF GitHub Bot commented on AIRFLOW-5702:
-
mik-laj commented on pull request #6372: [AIRFLOW-5702] Fix common docstring
issues
URL: https://github.com/apache/airflow/pull/6372
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix common docstring issues
> ---
>
> Key: AIRFLOW-5702
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5702
> Project: Apache Airflow
> Issue Type: Bug
> Components: documentation
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6372: [AIRFLOW-5702] Fix common docstring issues
mik-laj opened a new pull request #6372: [AIRFLOW-5702] Fix common docstring
issues
URL: https://github.com/apache/airflow/pull/6372
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5702) Fix common docstring issues
Kamil Bregula created AIRFLOW-5702: -- Summary: Fix common docstring issues Key: AIRFLOW-5702 URL: https://issues.apache.org/jira/browse/AIRFLOW-5702 Project: Apache Airflow Issue Type: Bug Components: documentation Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before execution
mik-laj commented on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before execution URL: https://github.com/apache/airflow/pull/6370#issuecomment-544144853 ``` Summary of failed tests tests.models.test_taskinstance.TestTaskInstance.test_xcom_pull_after_success Failure:builtins.AssertionError tests.utils.test_db.TestDb.test_database_schema_and_sqlalchemy_model_are_in_sync Failure:builtins.AssertionError ``` Travis is sad. Can you fix it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#discussion_r336736399
##
File path: airflow/operators/dagrun_operator.py
##
@@ -18,81 +18,64 @@
# under the License.
import datetime
-import json
-from typing import Callable, Dict, Optional, Union
+from typing import Dict, Optional, Union
from airflow.api.common.experimental.trigger_dag import trigger_dag
from airflow.models import BaseOperator
from airflow.utils import timezone
from airflow.utils.decorators import apply_defaults
-class DagRunOrder:
-def __init__(self, run_id=None, payload=None):
-self.run_id = run_id
-self.payload = payload
-
-
class TriggerDagRunOperator(BaseOperator):
"""
Triggers a DAG run for a specified ``dag_id``
:param trigger_dag_id: the dag_id to trigger (templated)
:type trigger_dag_id: str
-:param python_callable: a reference to a python function that will be
-called while passing it the ``context`` object and a placeholder
-object ``obj`` for your callable to fill and return if you want
-a DagRun created. This ``obj`` object contains a ``run_id`` and
-``payload`` attribute that you can modify in your function.
-The ``run_id`` should be a unique identifier for that DAG run, and
-the payload has to be a picklable object that will be made available
-to your tasks while executing that DAG run. Your function header
-should look like ``def foo(context, dag_run_obj):``
-:type python_callable: python callable
+:param conf: Configuration for the DAG run
+:type conf: dict
:param execution_date: Execution date for the dag (templated)
:type execution_date: str or datetime.datetime
"""
-template_fields = ('trigger_dag_id', 'execution_date')
-ui_color = '#ffefeb'
+
+template_fields = ("trigger_dag_id", "execution_date", "conf")
+ui_color = "#ffefeb"
@apply_defaults
def __init__(
-self,
-trigger_dag_id: str,
-python_callable: Optional[Callable[[Dict, DagRunOrder],
DagRunOrder]] = None,
-execution_date: Optional[Union[str, datetime.datetime]] = None,
-*args, **kwargs) -> None:
+self,
+trigger_dag_id: str,
+conf: Optional[Dict] = None,
+execution_date: Optional[Union[str, datetime.datetime]] = None,
+*args,
+**kwargs
+) -> None:
super().__init__(*args, **kwargs)
-self.python_callable = python_callable
self.trigger_dag_id = trigger_dag_id
+self.conf = conf
-self.execution_date = None # type: Optional[Union[str,
datetime.datetime]]
-if isinstance(execution_date, datetime.datetime):
-self.execution_date = execution_date.isoformat()
-elif isinstance(execution_date, str):
+if execution_date is None or isinstance(execution_date, (str,
datetime.datetime)):
self.execution_date = execution_date
-elif execution_date is None:
-self.execution_date = None
else:
raise TypeError(
-'Expected str or datetime.datetime type '
-'for execution_date. Got {}'.format(
-type(execution_date)))
+"Expected str or datetime.datetime type for execution_date. "
+"Got {}".format(type(execution_date))
+)
-def execute(self, context):
-if self.execution_date is not None:
-run_id = 'trig__{}'.format(self.execution_date)
-self.execution_date = timezone.parse(self.execution_date)
+def execute(self, context: Dict):
+if isinstance(self.execution_date, datetime.datetime):
+run_id = "trig__{}".format(self.execution_date.isoformat())
+elif isinstance(self.execution_date, str):
+run_id = "trig__{}".format(self.execution_date)
+self.execution_date = timezone.parse(self.execution_date) #
trigger_dag() expects datetime
Review comment:
Wouldn't it be better to set the execution date on line 58 where you are
setting it the first time?
```python
if isinstance(execution_date, str):
self.execution_date = timezone.parse(execution_date)
else:
self.execution_date = execution_date
```
IMO `self.execution_date = timezone.parse(self.execution_date)` is a kind of
validation so that should be made in the constructor even if it will be called
more often than in the `execute`.
then in the `execute` you can just do
```python
if self.execution_date is None:
self.execution_date = timezone.utcnow()
run_id = "trig__{}".format(self.execution_date.isoformat())
```
WDYT?
This is an automated message from the Apache Git Service.
To respond
[GitHub] [airflow] feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#discussion_r336736399
##
File path: airflow/operators/dagrun_operator.py
##
@@ -18,81 +18,64 @@
# under the License.
import datetime
-import json
-from typing import Callable, Dict, Optional, Union
+from typing import Dict, Optional, Union
from airflow.api.common.experimental.trigger_dag import trigger_dag
from airflow.models import BaseOperator
from airflow.utils import timezone
from airflow.utils.decorators import apply_defaults
-class DagRunOrder:
-def __init__(self, run_id=None, payload=None):
-self.run_id = run_id
-self.payload = payload
-
-
class TriggerDagRunOperator(BaseOperator):
"""
Triggers a DAG run for a specified ``dag_id``
:param trigger_dag_id: the dag_id to trigger (templated)
:type trigger_dag_id: str
-:param python_callable: a reference to a python function that will be
-called while passing it the ``context`` object and a placeholder
-object ``obj`` for your callable to fill and return if you want
-a DagRun created. This ``obj`` object contains a ``run_id`` and
-``payload`` attribute that you can modify in your function.
-The ``run_id`` should be a unique identifier for that DAG run, and
-the payload has to be a picklable object that will be made available
-to your tasks while executing that DAG run. Your function header
-should look like ``def foo(context, dag_run_obj):``
-:type python_callable: python callable
+:param conf: Configuration for the DAG run
+:type conf: dict
:param execution_date: Execution date for the dag (templated)
:type execution_date: str or datetime.datetime
"""
-template_fields = ('trigger_dag_id', 'execution_date')
-ui_color = '#ffefeb'
+
+template_fields = ("trigger_dag_id", "execution_date", "conf")
+ui_color = "#ffefeb"
@apply_defaults
def __init__(
-self,
-trigger_dag_id: str,
-python_callable: Optional[Callable[[Dict, DagRunOrder],
DagRunOrder]] = None,
-execution_date: Optional[Union[str, datetime.datetime]] = None,
-*args, **kwargs) -> None:
+self,
+trigger_dag_id: str,
+conf: Optional[Dict] = None,
+execution_date: Optional[Union[str, datetime.datetime]] = None,
+*args,
+**kwargs
+) -> None:
super().__init__(*args, **kwargs)
-self.python_callable = python_callable
self.trigger_dag_id = trigger_dag_id
+self.conf = conf
-self.execution_date = None # type: Optional[Union[str,
datetime.datetime]]
-if isinstance(execution_date, datetime.datetime):
-self.execution_date = execution_date.isoformat()
-elif isinstance(execution_date, str):
+if execution_date is None or isinstance(execution_date, (str,
datetime.datetime)):
self.execution_date = execution_date
-elif execution_date is None:
-self.execution_date = None
else:
raise TypeError(
-'Expected str or datetime.datetime type '
-'for execution_date. Got {}'.format(
-type(execution_date)))
+"Expected str or datetime.datetime type for execution_date. "
+"Got {}".format(type(execution_date))
+)
-def execute(self, context):
-if self.execution_date is not None:
-run_id = 'trig__{}'.format(self.execution_date)
-self.execution_date = timezone.parse(self.execution_date)
+def execute(self, context: Dict):
+if isinstance(self.execution_date, datetime.datetime):
+run_id = "trig__{}".format(self.execution_date.isoformat())
+elif isinstance(self.execution_date, str):
+run_id = "trig__{}".format(self.execution_date)
+self.execution_date = timezone.parse(self.execution_date) #
trigger_dag() expects datetime
Review comment:
Wouldn't it be better to set the execution date on line 58 where you are
setting it the first time?
```python
if isinstance(execution_date, str):
self.execution_date = timezone.parse(self.execution_date)
else:
self.execution_date = execution_date
```
IMO `self.execution_date = timezone.parse(self.execution_date)` is a kind of
validation so that should be made in the constructor even if it will be called
more often than in the `execute`.
then in the `execute` you can just do
```python
if self.execution_date is None:
self.execution_date = timezone.utcnow()
run_id = "trig__{}".format(self.execution_date.isoformat())
```
WDYT?
This is an automated message from the Apache Git Service.
To
[GitHub] [airflow] feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#discussion_r334251729
##
File path: airflow/example_dags/example_trigger_target_dag.py
##
@@ -18,65 +18,36 @@
# under the License.
"""
-This example illustrates the use of the TriggerDagRunOperator. There are 2
-entities at work in this scenario:
-1. The Controller DAG - the DAG that conditionally executes the trigger
- (in example_trigger_controller.py)
-2. The Target DAG - DAG being triggered
-
-This example illustrates the following features :
-1. A TriggerDagRunOperator that takes:
- a. A python callable that decides whether or not to trigger the Target DAG
- b. An optional params dict passed to the python callable to help in
- evaluating whether or not to trigger the Target DAG
- c. The id (name) of the Target DAG
- d. The python callable can add contextual info to the DagRun created by
- way of adding a Pickleable payload (e.g. dictionary of primitives). This
- state is then made available to the TargetDag
-2. A Target DAG : c.f. example_trigger_target_dag.py
+Example usage of the TriggerDagRunOperator. This example holds 2 DAGs:
+1. 1st DAG (example_trigger_controller_dag) holds a TriggerDagRunOperator,
which will trigger the 2nd DAG
+2. 2nd DAG (example_trigger_target_dag) which will be triggered by the
TriggerDagRunOperator in the 1st DAG
"""
-import pprint
-
-import airflow
+import airflow.utils.dates
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
-pp = pprint.PrettyPrinter(indent=4)
-
-args = {
-'start_date': airflow.utils.dates.days_ago(2),
-'owner': 'Airflow',
-}
-
dag = DAG(
-dag_id='example_trigger_target_dag',
-default_args=args,
+dag_id="example_trigger_target_dag",
+default_args={"start_date": airflow.utils.dates.days_ago(2), "owner":
"Airflow"},
schedule_interval=None,
)
-def run_this_func(**kwargs):
+def run_this_func(**context):
"""
Print the payload "message" passed to the DagRun conf attribute.
:param dict kwargs: Context
Review comment:
```suggestion
:param context: The execution context
:type context: dict
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
feluelle commented on a change in pull request #6317: [AIRFLOW-5644] Simplify
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#discussion_r336736399
##
File path: airflow/operators/dagrun_operator.py
##
@@ -18,81 +18,64 @@
# under the License.
import datetime
-import json
-from typing import Callable, Dict, Optional, Union
+from typing import Dict, Optional, Union
from airflow.api.common.experimental.trigger_dag import trigger_dag
from airflow.models import BaseOperator
from airflow.utils import timezone
from airflow.utils.decorators import apply_defaults
-class DagRunOrder:
-def __init__(self, run_id=None, payload=None):
-self.run_id = run_id
-self.payload = payload
-
-
class TriggerDagRunOperator(BaseOperator):
"""
Triggers a DAG run for a specified ``dag_id``
:param trigger_dag_id: the dag_id to trigger (templated)
:type trigger_dag_id: str
-:param python_callable: a reference to a python function that will be
-called while passing it the ``context`` object and a placeholder
-object ``obj`` for your callable to fill and return if you want
-a DagRun created. This ``obj`` object contains a ``run_id`` and
-``payload`` attribute that you can modify in your function.
-The ``run_id`` should be a unique identifier for that DAG run, and
-the payload has to be a picklable object that will be made available
-to your tasks while executing that DAG run. Your function header
-should look like ``def foo(context, dag_run_obj):``
-:type python_callable: python callable
+:param conf: Configuration for the DAG run
+:type conf: dict
:param execution_date: Execution date for the dag (templated)
:type execution_date: str or datetime.datetime
"""
-template_fields = ('trigger_dag_id', 'execution_date')
-ui_color = '#ffefeb'
+
+template_fields = ("trigger_dag_id", "execution_date", "conf")
+ui_color = "#ffefeb"
@apply_defaults
def __init__(
-self,
-trigger_dag_id: str,
-python_callable: Optional[Callable[[Dict, DagRunOrder],
DagRunOrder]] = None,
-execution_date: Optional[Union[str, datetime.datetime]] = None,
-*args, **kwargs) -> None:
+self,
+trigger_dag_id: str,
+conf: Optional[Dict] = None,
+execution_date: Optional[Union[str, datetime.datetime]] = None,
+*args,
+**kwargs
+) -> None:
super().__init__(*args, **kwargs)
-self.python_callable = python_callable
self.trigger_dag_id = trigger_dag_id
+self.conf = conf
-self.execution_date = None # type: Optional[Union[str,
datetime.datetime]]
-if isinstance(execution_date, datetime.datetime):
-self.execution_date = execution_date.isoformat()
-elif isinstance(execution_date, str):
+if execution_date is None or isinstance(execution_date, (str,
datetime.datetime)):
self.execution_date = execution_date
-elif execution_date is None:
-self.execution_date = None
else:
raise TypeError(
-'Expected str or datetime.datetime type '
-'for execution_date. Got {}'.format(
-type(execution_date)))
+"Expected str or datetime.datetime type for execution_date. "
+"Got {}".format(type(execution_date))
+)
-def execute(self, context):
-if self.execution_date is not None:
-run_id = 'trig__{}'.format(self.execution_date)
-self.execution_date = timezone.parse(self.execution_date)
+def execute(self, context: Dict):
+if isinstance(self.execution_date, datetime.datetime):
+run_id = "trig__{}".format(self.execution_date.isoformat())
+elif isinstance(self.execution_date, str):
+run_id = "trig__{}".format(self.execution_date)
+self.execution_date = timezone.parse(self.execution_date) #
trigger_dag() expects datetime
Review comment:
Wouldn't it be better to set the execution date on line 58 where you are
setting it the first time?
```python
if isinstance(execution_date, str):
self.execution_date = timezone.parse(self.execution_date)
else:
self.execution_date = execution_date
```
IMO `self.execution_date = timezone.parse(self.execution_date)` is a kind of
validation so that should be made in the constructor even if it will be called
more often than in the `execute`.
then you can just do
```python
if self.execution_date is None:
self.execution_date = timezone.utcnow()
run_id = "trig__{}".format(self.execution_date.isoformat())
```
WDYT?
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [airflow] BasPH commented on a change in pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
BasPH commented on a change in pull request #6317: [AIRFLOW-5644] Simplify
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#discussion_r336735481
##
File path: airflow/operators/dagrun_operator.py
##
@@ -75,24 +63,21 @@ def __init__(
self.execution_date = None
else:
raise TypeError(
-'Expected str or datetime.datetime type '
-'for execution_date. Got {}'.format(
-type(execution_date)))
+"Expected str or datetime.datetime type for execution_date. "
+"Got {}".format(type(execution_date))
+)
def execute(self, context):
Review comment:
The `__init__` arguments are already type annotated. Personally not a fan of
these `if/else` type checks everywhere, it's against Python's nature :-) Do
you think we should remove it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-5623) latest_only_operator fails for schedule_interval='@once'
[
https://issues.apache.org/jira/browse/AIRFLOW-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Winkel reassigned AIRFLOW-5623:
--
Assignee: (was: Martin Winkel)
> latest_only_operator fails for schedule_interval='@once'
>
>
> Key: AIRFLOW-5623
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5623
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.5
>Reporter: Gerben Oostra
>Priority: Minor
>
> Observation: In a dag with schedule_interval set to @once, the
> `latest_only_operator` fails with the following error:
>
> {{[2019-10-09 09:51:37,346] \{latest_only_operator.py:48} INFO - Checking
> latest only with left_window: None right_window: None now: 2019-10-09
> 07:51:37.346697+00:00
> [2019-10-09 09:51:37,347] \{models.py:1736} ERROR - '<' not supported between
> instances of 'NoneType' and 'datetime.datetime'
> Traceback (most recent call last):
> File
> "//anaconda/envs/airflow/lib/python3.6/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "//anaconda/envs/airflow/lib/python3.6/site-packages/airflow/operators/latest_only_operator.py",
> line 51, in execute
> if not left_window < now <= right_window:
> TypeError: '<' not supported between instances of 'NoneType' and
> 'datetime.datetime'
> [2019-10-09 09:51:37,363] \{models.py:1756} INFO - Marking task as
> UP_FOR_RETRY}}
> I expected it to succeed, and allow the remainder of the dag to be ran. (if
> an @once dag is running, it is always the latest)
> Rootcause analysis:
> If the `schedule_interval` of a dag is `@once`, the dag's field
> `self._schedule_interval` is set to `None`.
> The `latest_only_operator` determines the window by passing the execution
> date to the dags `following_schedule()`. There the dag's
> `self._schedule_interval` type is compared to `six.string_types` and
> `timedelta`. Both type checks fail, so nothing (`None`) is returned.
> Causing the time window comparison to fail.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
