[GitHub] Fokko commented on issue #4018: [AIRFLOW-3195] Log query and task_id in druid-hook
Fokko commented on issue #4018: [AIRFLOW-3195] Log query and task_id in druid-hook URL: https://github.com/apache/incubator-airflow/pull/4018#issuecomment-429304568 Nice one @awelsh93. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #4018: [AIRFLOW-3195] Log query and task_id in druid-hook
Fokko closed pull request #4018: [AIRFLOW-3195] Log query and task_id in
druid-hook
URL: https://github.com/apache/incubator-airflow/pull/4018
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/hooks/druid_hook.py b/airflow/hooks/druid_hook.py
index 75111fad2e..a45ebf472f 100644
--- a/airflow/hooks/druid_hook.py
+++ b/airflow/hooks/druid_hook.py
@@ -68,6 +68,7 @@ def get_conn_url(self):
def submit_indexing_job(self, json_index_spec):
url = self.get_conn_url()
+self.log.info("Druid ingestion spec: {}".format(json_index_spec))
req_index = requests.post(url, json=json_index_spec,
headers=self.header)
if req_index.status_code != 200:
raise AirflowException('Did not get 200 when '
@@ -76,6 +77,7 @@ def submit_indexing_job(self, json_index_spec):
req_json = req_index.json()
# Wait until the job is completed
druid_task_id = req_json['task']
+self.log.info("Druid indexing task-id: {}".format(druid_task_id))
running = True
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko closed pull request #4033: [AIRFLOW-3187] Update airflow.gif file with a slower version
Fokko closed pull request #4033: [AIRFLOW-3187] Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/img/airflow.gif b/docs/img/airflow.gif index 1889b868f7..7646e37f35 100644 Binary files a/docs/img/airflow.gif and b/docs/img/airflow.gif differ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4033: [AIRFLOW-3187] Update airflow.gif file with a slower version
Fokko commented on issue #4033: [AIRFLOW-3187] Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033#issuecomment-429305371 Thanks @PaulVelthuis93 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
codecov-io edited a comment on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#issuecomment-426543786 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=h1) Report > Merging [#3989](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr&el=desc) will **decrease** coverage by `2.91%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3989 +/- ## == - Coverage 75.79% 72.87% -2.92% == Files 199 199 Lines 1594617003+1057 == + Hits1208612391 +305 - Misses 3860 4612 +752 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `58.76% <0%> (-5.72%)` | :arrow_down: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `67.36% <0%> (-20.31%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `60.97% <0%> (-18.69%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `75.55% <0%> (-16.16%)` | :arrow_down: | | [airflow/utils/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jb25maWd1cmF0aW9uLnB5) | `85.71% <0%> (-14.29%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `85.07% <0%> (-4.19%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3989/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.18% <0%> (+0.14%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=footer). Last update [b8be322...0d1ed47](https://codecov.io/gh/apache/incubator-airflow/pull/3989?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
Fokko commented on a change in pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038#discussion_r224764734 ## File path: airflow/models.py ## @@ -150,6 +150,10 @@ def get_fernet(): :raises: AirflowException if there's a problem trying to load Fernet """ global _fernet +log = LoggingMixin().log +no_encryption_phrase = \ Review comment: Ah I see, returning `None` instead of `cryptography_not_found_storing_passwords_in_plain_text` would be my preference as well This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] deagon opened a new pull request #4042: [AIRFLOW-3193] Pin docker requirement version
deagon opened a new pull request #4042: [AIRFLOW-3193] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/4042 The method "create_container" in APIClient of docker has been incompatible from version 3.0.0. ### Jira - https://issues.apache.org/jira/browse/AIRFLOW-3193 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] deagon commented on issue #3975: [AIRFLOW-XXX] Pin docker requirement version
deagon commented on issue #3975: [AIRFLOW-XXX] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/3975#issuecomment-429312236 Reopened it at #4042 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4042: [AIRFLOW-3193] Pin docker requirement version
codecov-io edited a comment on issue #4042: [AIRFLOW-3193] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/4042#issuecomment-429322410 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=h1) Report > Merging [#4042](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4042 +/- ## == - Coverage 75.82% 75.82% -0.01% == Files 199 199 Lines 1594915949 == - Hits1209412093 -1 - Misses 3855 3856 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4042/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=footer). Last update [7ae0fe2...e32a30e](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4042: [AIRFLOW-3193] Pin docker requirement version
codecov-io commented on issue #4042: [AIRFLOW-3193] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/4042#issuecomment-429322410 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=h1) Report > Merging [#4042](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4042 +/- ## == - Coverage 75.82% 75.82% -0.01% == Files 199 199 Lines 1594915949 == - Hits1209412093 -1 - Misses 3855 3856 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4042/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=footer). Last update [7ae0fe2...e32a30e](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4042: [AIRFLOW-3193] Pin docker requirement version
codecov-io edited a comment on issue #4042: [AIRFLOW-3193] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/4042#issuecomment-429322410 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=h1) Report > Merging [#4042](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4042 +/- ## == - Coverage 75.82% 75.82% -0.01% == Files 199 199 Lines 1594915949 == - Hits1209412093 -1 - Misses 3855 3856 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4042/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=footer). Last update [7ae0fe2...e32a30e](https://codecov.io/gh/apache/incubator-airflow/pull/4042?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kbl commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
kbl commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-429325158 What @Fokko mentioned about session leakage could be true. I've deployed that a while ago as a patch on top of 1.10. We're running Airflow with 2 airworkers x 12 celery processors on each node. We've seen an increase in number of open connections to our database after the upgrade from 1.9 to 1.10+patch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] exploy commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
exploy commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224783788 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -205,10 +206,18 @@ def __init__(self, self.idle_delete_ttl = idle_delete_ttl self.auto_delete_time = auto_delete_time self.auto_delete_ttl = auto_delete_ttl +self.single_node = num_workers == 0 assert not (self.custom_image and self.image_version), \ "custom_image and image_version can't be both set" +assert ( Review comment: I can imagine someone trying to setup cluster which consists only preemptible workers (`num_preemptible_workers`) and no regular workers (`num_workers`). Generally, we could assert on `num_workers` and `num_preemptible_workers` and allow only cases: - both values are zeros - `num_workers` >= 2 WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant
codecov-io commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035#issuecomment-429326436 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=h1) Report > Merging [#4035](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `63.63%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4035 +/- ## == + Coverage 75.82% 75.84% +0.01% == Files 199 199 Lines 1594915949 == + Hits1209412097 +3 + Misses 3855 3852 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.6% <ø> (ø)` | :arrow_up: | | [airflow/utils/trigger\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90cmlnZ2VyX3J1bGUucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `86.11% <ø> (ø)` | :arrow_up: | | [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `81.42% <100%> (ø)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <100%> (ø)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.48% <50%> (+0.29%)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.85% <50%> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=footer). Last update [7ae0fe2...c7c2842](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4035: [AIRFLOW-3190] Make flake8 compliant
codecov-io edited a comment on issue #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035#issuecomment-429326436 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=h1) Report > Merging [#4035](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `63.63%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4035 +/- ## == + Coverage 75.82% 75.84% +0.01% == Files 199 199 Lines 1594915949 == + Hits1209412097 +3 + Misses 3855 3852 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.6% <ø> (ø)` | :arrow_up: | | [airflow/utils/trigger\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90cmlnZ2VyX3J1bGUucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `86.11% <ø> (ø)` | :arrow_up: | | [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `81.42% <100%> (ø)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <100%> (ø)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.48% <50%> (+0.29%)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.85% <50%> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4035/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=footer). Last update [7ae0fe2...c7c2842](https://codecov.io/gh/apache/incubator-airflow/pull/4035?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant
Fokko commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035#issuecomment-429326612 @kaxil All green, hit the button! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4040: [AIRFLOW-3192] Remove deprecated post_execute logic
codecov-io commented on issue #4040: [AIRFLOW-3192] Remove deprecated post_execute logic URL: https://github.com/apache/incubator-airflow/pull/4040#issuecomment-429327134 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=h1) Report > Merging [#4040](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4040 +/- ## == + Coverage 75.82% 75.83% +<.01% == Files 199 199 Lines 1594915943 -6 == - Hits1209412091 -3 + Misses 3855 3852 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4040/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.82% <100%> (+0.1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=footer). Last update [7ae0fe2...5dcc74b](https://codecov.io/gh/apache/incubator-airflow/pull/4040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429043686 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=h1) Report > Merging [#4036](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `0.05%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4036 +/- ## == + Coverage 75.82% 75.87% +0.05% == Files 199 199 Lines 1594915966 +17 == + Hits1209412115 +21 + Misses 3855 3851 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `91.95% <100%> (+0.59%)` | :arrow_up: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `97.05% <100%> (+0.35%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.48% <0%> (+0.35%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=footer). Last update [7ae0fe2...0408e6d](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp commented on issue #3984: [AIRFLOW-3141] Handle duration for missing dag.
jmcarp commented on issue #3984: [AIRFLOW-3141] Handle duration for missing dag. URL: https://github.com/apache/incubator-airflow/pull/3984#issuecomment-429334209 Updated the rbac duration view. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224792403 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -205,10 +206,18 @@ def __init__(self, self.idle_delete_ttl = idle_delete_ttl self.auto_delete_time = auto_delete_time self.auto_delete_ttl = auto_delete_ttl +self.single_node = num_workers == 0 assert not (self.custom_image and self.image_version), \ "custom_image and image_version can't be both set" +assert ( Review comment: Oh, that is true! ;) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oliviersm199 commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
oliviersm199 commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429334447 Yes merging in master helped fix the code coverage report! Thank you @Fokko. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4040: [AIRFLOW-3192] Remove deprecated post_execute logic
Fokko commented on a change in pull request #4040: [AIRFLOW-3192] Remove deprecated post_execute logic URL: https://github.com/apache/incubator-airflow/pull/4040#discussion_r224796215 ## File path: airflow/models.py ## @@ -1650,22 +1650,6 @@ def signal_handler(signum, frame): if result is not None: self.xcom_push(key=XCOM_RETURN_KEY, value=result) -# TODO remove deprecated behavior in Airflow 2.0 -try: -task_copy.post_execute(context=context, result=result) Review comment: Whoops, too fast, thanks @ashb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#issuecomment-429340391 That's a valid concern, but my concern is that it will be much worse if airflow has to return all the runs and filtering has to be done on the caller's side. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#issuecomment-429342495 Rebasing on to master should fix the tests, hopefully. I think given this is an optional feature there is no penalty for anyone who doesn't use this, so sounds good to me. One thought: instead of just prefix is it worth changing the parameter to be `run_id__like` which allows for more flexability (and double `__` to follow the Django ORM convention of separating the field from the operator) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dossett commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
dossett commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224801183 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -170,6 +172,7 @@ def __init__(self, idle_delete_ttl=None, auto_delete_time=None, auto_delete_ttl=None, + single_node=False, Review comment: Love the suggestion and the change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#discussion_r224800834 ## File path: airflow/models.py ## @@ -4860,6 +4860,8 @@ def find(dag_id=None, run_id=None, execution_date=None, :param run_id: defines the the run id for this dag run :type run_id: string :param execution_date: the execution date +:param run_id_prefix: match runs with the given prefix in the run_id. +:type run_id_prefix: string Review comment: Ordering - you've split execution_date desc from it's type This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#discussion_r224801246 ## File path: airflow/models.py ## @@ -4878,6 +4880,8 @@ def find(dag_id=None, run_id=None, execution_date=None, qry = qry.filter(DR.dag_id == dag_id) if run_id: qry = qry.filter(DR.run_id == run_id) +if run_id_prefix: Review comment: This one should probably be an `elif` as specifying run_id and run_id_prefix together doesn't make much sense This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#discussion_r224801083 ## File path: airflow/models.py ## @@ -4849,7 +4849,7 @@ def refresh_from_db(self, session=None): @staticmethod @provide_session -def find(dag_id=None, run_id=None, execution_date=None, +def find(dag_id=None, run_id=None, execution_date=None, run_id_prefix=None, Review comment: Please add run_id_prefix (or whatever we decide to call the new param) to almost the end of the list, just before `session` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#discussion_r224801553 ## File path: airflow/models.py ## @@ -5154,6 +5158,19 @@ def get_latest_runs(cls, session): ) return dagruns +@classmethod +@provide_session +def get_runs(cls, dag_id, run_id_prefix, session=None): Review comment: Nothing calls this method does it? At the very least if it's useful to you then we need tests covering it, and more detailed docstring. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3984: [AIRFLOW-3141] Handle duration for missing dag.
codecov-io edited a comment on issue #3984: [AIRFLOW-3141] Handle duration for missing dag. URL: https://github.com/apache/incubator-airflow/pull/3984#issuecomment-426309110 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=h1) Report > Merging [#3984](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3984 +/- ## == + Coverage 75.82% 75.83% +<.01% == Files 199 199 Lines 1594915955 +6 == + Hits1209412100 +6 Misses 3855 3855 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.91% <100%> (+0.05%)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.25% <100%> (+0.06%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=footer). Last update [7ae0fe2...8af745a](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3984: [AIRFLOW-3141] Handle duration for missing dag.
codecov-io edited a comment on issue #3984: [AIRFLOW-3141] Handle duration for missing dag. URL: https://github.com/apache/incubator-airflow/pull/3984#issuecomment-426309110 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=h1) Report > Merging [#3984](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7ae0fe2bb49787f07adb0f08907454acda3728f6?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3984 +/- ## == + Coverage 75.82% 75.83% +<.01% == Files 199 199 Lines 1594915955 +6 == + Hits1209412100 +6 Misses 3855 3855 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.91% <100%> (+0.05%)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3984/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.25% <100%> (+0.06%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=footer). Last update [7ae0fe2...8af745a](https://codecov.io/gh/apache/incubator-airflow/pull/3984?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster
Fokko commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#issuecomment-429351174 Thanks @exploy Merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
Fokko closed pull request #4015: [AIRFLOW-2789] Create single node DataProc
cluster
URL: https://github.com/apache/incubator-airflow/pull/4015
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/operators/dataproc_operator.py
b/airflow/contrib/operators/dataproc_operator.py
index 49e24a3df2..af2a211539 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -52,7 +52,8 @@ class DataprocClusterCreateOperator(BaseOperator):
:param project_id: The ID of the google cloud project in which
to create the cluster. (templated)
:type project_id: str
-:param num_workers: The # of workers to spin up
+:param num_workers: The # of workers to spin up. If set to zero will
+spin up cluster in a single node mode
:type num_workers: int
:param storage_bucket: The storage bucket to use, setting to None lets
dataproc
generate a custom one for you
@@ -186,7 +187,7 @@ def __init__(self,
self.metadata = metadata
self.custom_image = custom_image
self.image_version = image_version
-self.properties = properties
+self.properties = properties or dict()
self.master_machine_type = master_machine_type
self.master_disk_type = master_disk_type
self.master_disk_size = master_disk_size
@@ -205,10 +206,18 @@ def __init__(self,
self.idle_delete_ttl = idle_delete_ttl
self.auto_delete_time = auto_delete_time
self.auto_delete_ttl = auto_delete_ttl
+self.single_node = num_workers == 0
assert not (self.custom_image and self.image_version), \
"custom_image and image_version can't be both set"
+assert (
+not self.single_node or (
+self.single_node and self.num_preemptible_workers == 0
+)
+), "num_workers == 0 means single node mode - no preemptibles allowed"
+
+
def _get_cluster_list_for_project(self, service):
result = service.projects().regions().clusters().list(
projectId=self.project_id,
@@ -351,7 +360,12 @@ def _build_cluster_data(self):
'{}/global/images/{}'.format(self.project_id,
self.custom_image)
cluster_data['config']['masterConfig']['imageUri'] =
custom_image_url
-cluster_data['config']['workerConfig']['imageUri'] =
custom_image_url
+if not self.single_node:
+cluster_data['config']['workerConfig']['imageUri'] =
custom_image_url
+
+if self.single_node:
+self.properties["dataproc:dataproc.allow.zero.workers"] = "true"
+
if self.properties:
cluster_data['config']['softwareConfig']['properties'] =
self.properties
if self.idle_delete_ttl:
diff --git a/tests/contrib/operators/test_dataproc_operator.py
b/tests/contrib/operators/test_dataproc_operator.py
index 60c1268ee7..fb90606ea5 100644
--- a/tests/contrib/operators/test_dataproc_operator.py
+++ b/tests/contrib/operators/test_dataproc_operator.py
@@ -298,6 +298,34 @@ def test_init_with_custom_image(self):
self.assertEqual(cluster_data['config']['workerConfig']['imageUri'],
expected_custom_image_url)
+def test_build_single_node_cluster(self):
+dataproc_operator = DataprocClusterCreateOperator(
+task_id=TASK_ID,
+cluster_name=CLUSTER_NAME,
+project_id=PROJECT_ID,
+num_workers=0,
+num_preemptible_workers=0,
+zone=ZONE,
+dag=self.dag
+)
+cluster_data = dataproc_operator._build_cluster_data()
+self.assertEqual(
+cluster_data['config']['softwareConfig']['properties']
+['dataproc:dataproc.allow.zero.workers'], "true")
+
+def test_init_cluster_with_zero_workers_and_not_non_zero_preemtibles(self):
+with self.assertRaises(AssertionError):
+DataprocClusterCreateOperator(
+task_id=TASK_ID,
+cluster_name=CLUSTER_NAME,
+project_id=PROJECT_ID,
+num_workers=0,
+num_preemptible_workers=2,
+zone=ZONE,
+dag=self.dag,
+image_version=IMAGE_VERSION,
+)
+
def test_cluster_name_log_no_sub(self):
with patch('airflow.contrib.operators.dataproc_operator.DataProcHook')
\
as mock_hook:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above t
[GitHub] rcorre commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
rcorre commented on a change in pull request #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#discussion_r224817073 ## File path: airflow/models.py ## @@ -5154,6 +5158,19 @@ def get_latest_runs(cls, session): ) return dagruns +@classmethod +@provide_session +def get_runs(cls, dag_id, run_id_prefix, session=None): Review comment: Nope, realized I could do it more easily by adding a param to `find` and forgot to remove this This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG closed pull request #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators
XD-DENG closed pull request #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#issuecomment-429358961 Thanks for the comments! I addressed all of them in a separate commit for ease of review. I can squash before merging if that's the protocol in this repo. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb edited a comment on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
ashb edited a comment on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API. URL: https://github.com/apache/incubator-airflow/pull/3546#issuecomment-429342495 Rebasing on to master should fix the tests, hopefully. I think given this is an optional feature there is no penalty for anyone who doesn't use this, so sounds good to me. One thought: instead of just prefix is it worth changing the parameter to be `run_id__like` which allows for more flexability (and double `__` to follow the Django ORM convention of separating the field from the operator) - `run_id_prefix=test` would become `run_id__like=test%`. Thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG opened a new pull request #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks
XD-DENG opened a new pull request #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks URL: https://github.com/apache/incubator-airflow/pull/4043 ### JIRA https://issues.apache.org/jira/browse/AIRFLOW-3174 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: - These SQL operators (based on `DbApiHook`) receive `sql` and `parameters` (optional) as arguments. But `parameters` were not documented. The contents I added is based on the docstring in https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/dbapi_hook.py#L88 - Fixed two minor style errors which would cause incorrect documentation rendering - Added docstring for `autocommit` parameter in relevant operators. The content is based on https://db.apache.org/derby/docs/10.9/devguide/cdevconcepts29416.html and https://www.postgresql.org/docs/9.1/static/ecpg-commands.html - Adjusted order or docstring items to make sure they are consistent with the order of arguments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks
XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks URL: https://github.com/apache/incubator-airflow/pull/4043#issuecomment-429368832 Hi @Fokko , I accidentally screwed my branch and could not push to my earlier PR https://github.com/apache/incubator-airflow/pull/4023 . So I had to open this new PR. The comments you raised earlier (adjust order for consistency, add docstring for `autocommit`) are addressed. Kindly have a check. I also addressed two minor style errors in two Hook files. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks
XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks URL: https://github.com/apache/incubator-airflow/pull/4043#issuecomment-429369887 BTW, given it's only docstring change, I had `[ci skip]` in my commit message to skip CI tests. I had already run Flake8 check locally to make sure no Flake8 error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb opened a new pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters off the AWS API
ashb opened a new pull request #4044: [AIRFLOW-3197] EMRHook is missing new
parameters off the AWS API
URL: https://github.com/apache/incubator-airflow/pull/4044
Make sure you have checked _all_ steps below.
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-3197
### Description
- [x] Allow passing any params to the CreateJobFlow API, so that we don't
have
to stay up to date with AWS api changes.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Eronarn commented on a change in pull request #3584: [AIRFLOW-249] Refactor the SLA mechanism
Eronarn commented on a change in pull request #3584: [AIRFLOW-249] Refactor the SLA mechanism URL: https://github.com/apache/incubator-airflow/pull/3584#discussion_r224850526 ## File path: airflow/migrations/versions/19e9c503bc4e_add_sla_miss_type.py ## @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Add SLA miss type. + +Revision ID: 19e9c503bc4e +Revises: 0e2a74e0fc9f +Create Date: 2018-07-05 12:19:18.718960 + +""" + +# revision identifiers, used by Alembic. +revision = '19e9c503bc4e' +down_revision = '0e2a74e0fc9f' +branch_labels = None +depends_on = None + +from alembic import op +import sqlalchemy as sa + + +SLA_MISS_TABLE = "sla_miss" +NEW_COLUMN = "sla_type" +OLD_SLA_MISS_TYPE = "task_late_finish" + + +def upgrade(): +op.add_column(SLA_MISS_TABLE, + sa.Column( + NEW_COLUMN, + sa.String(50), + primary_key=True, + nullable=False, + server_default=OLD_SLA_MISS_TYPE + )) + + +def downgrade(): +op.drop_column(SLA_MISS_TABLE, NEW_COLUMN) Review comment: I don't really know enough about migrations to know how to implement/test this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Eronarn commented on a change in pull request #3584: [AIRFLOW-249] Refactor the SLA mechanism
Eronarn commented on a change in pull request #3584: [AIRFLOW-249] Refactor the
SLA mechanism
URL: https://github.com/apache/incubator-airflow/pull/3584#discussion_r224851184
##
File path: airflow/utils/sla.py
##
@@ -0,0 +1,371 @@
+import airflow.models
+from airflow.utils import asciiart
+from airflow.utils.db import provide_session
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.state import State
+
+log = LoggingMixin().log
+
+
+def yield_unscheduled_runs(dag, last_scheduled_run, ts):
+"""
+Yield new DagRuns that haven't been created yet.
+"""
+
+# TODO: A lot of this logic is duplicated from the scheduler. It would
+# be better to have one function that yields upcoming DAG runs in a
+# consistent way that is usable for both use cases.
+
+# Start by assuming that there is no next run.
+next_run_date = None
+
+# The first DAGRun has not been created yet.
+if not last_scheduled_run:
+task_start_dates = [t.start_date for t in dag.tasks]
+if task_start_dates:
+next_run_date = dag.normalize_schedule(min(task_start_dates))
+# The DagRun is @once and has already happened.
+elif dag.schedule_interval == '@once':
+return
+# Start from the next "normal" run.
+else:
+next_run_date =
dag.following_schedule(last_scheduled_run.execution_date)
+
+while True:
+# There should be a next execution.
+if not next_run_date:
+return
+
+# The next execution shouldn't be in the future.
+if next_run_date > ts:
+return
+
+# The next execution shouldn't be beyond the DAG's end date.
+# n.b. - tasks have their own end dates checked later
+if next_run_date and dag.end_date and next_run_date > dag.end_date:
+return
+
+# Calculate the end of this execution period.
+if dag.schedule_interval == '@once':
+period_end = next_run_date
+else:
+period_end = dag.following_schedule(next_run_date)
+
+# The next execution shouldn't still be mid-period.
+if period_end > ts:
+return
+
+# We've passed every filter; this is a valid future DagRun that
+# presumably hasn't been scheduled due to concurrency limits.
+# Create a DAGRun, though it won't exist in the db yet.
+next_run = dag.create_dagrun(
+run_id=airflow.models.DagRun.ID_PREFIX + next_run_date.isoformat(),
+execution_date=next_run_date,
+start_date=ts,
+state=State.RUNNING,
+external_trigger=False
+)
+yield next_run
+
+# Examine the next date.
+next_run_date = dag.following_schedule(next_run_date)
+
+
+def yield_unscheduled_tis(dag_run, ts, session=None):
+"""
+Given an unscheduled `DagRun`, yield any unscheduled TIs that will exist
+for it in the future, respecting the end date of the DAG and task.
+"""
+for task in dag_run.dag.tasks:
+end_dates = []
+if dag_run.dag.end_date:
+end_dates.append(dag_run.dag.end_date)
+if task.end_date:
+end_dates.append(task.end_date)
+
+# Create TIs if there is no end date, or it hasn't happened yet.
+if not end_dates or ts < min(end_dates):
+yield airflow.models.TaskInstance(task, dag_run.execution_date)
+
+
+def get_sla_misses(ti, session):
+"""
+Get all SLA misses that match a particular TaskInstance.
+"""
+SM = airflow.models.SlaMiss
+return session.query(SM).filter(
+SM.dag_id == ti.dag_id,
+SM.task_id == ti.task_id,
+SM.execution_date == ti.execution_date
+).all()
+
+
+def create_sla_misses(ti, ts, session):
+"""
+Determine whether a TaskInstance has missed any SLAs as of a provided
+timestamp. If it has, create `SlaMiss` objects in the provided session.
+Note that one TaskInstance can have multiple SLA miss objects: for example,
+it can both start late and run longer than expected.
+"""
+# Skipped task instances will never trigger SLAs because they
+# were intentionally not scheduled. Though, it's still a valid and
+# interesting SLA miss if a task that's *going* to be skipped today is
+# late! That could mean that an upstream task is hanging.
+if ti.state == State.SKIPPED:
+return
+
+SM = airflow.models.SlaMiss
+
+# Get existing misses.
+ti_misses = {sm.sla_type: sm for sm in get_sla_misses(ti, session)}
+
+# Calculate each type of SLA miss. Wrapping exceptions here is
+# important so that an exception in one type of SLA doesn't
+# prevent other task SLAs from getting triggered.
+
+# SLA Miss for Expected Duration
+if SM.TASK_DURATION_EXCEEDED not in ti_misses \
+and ti.task.expected_duration and ti.start_date:
+try:
+
[GitHub] tswast commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table
tswast commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table URL: https://github.com/apache/incubator-airflow/pull/4003#issuecomment-429394476 I think I agree that this isn't general enough to warrant a whole operator. It would mean an explosion of operators if we provide one for every property of every resource. Perhaps a "patch BigQuery table metadata" operator would be more appropriate? It could accept a dictionary of properties to update on the table resource. See: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/patch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tswast edited a comment on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table
tswast edited a comment on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table URL: https://github.com/apache/incubator-airflow/pull/4003#issuecomment-429394476 I agree that this isn't general enough to warrant a whole operator. It would mean an explosion of operators if we provide one for every property of every resource. Perhaps a "patch BigQuery table metadata" operator would be more appropriate? It could accept a dictionary of properties to update on the table resource. See: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/patch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Eronarn commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism
Eronarn commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism URL: https://github.com/apache/incubator-airflow/pull/3584#issuecomment-429396092 I believe I've addressed most of the above feedback successfully. There's still some issues, but I am working on this actively again, with a focus on improving documentation/UX and adding testing. For now, I am pushing all commits to this branch, since it makes it a bit easier to see in-progress development as I continue to push. I will squash this branch into a single commit once I get a green light to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3584: [AIRFLOW-249] Refactor the SLA mechanism
ashb commented on a change in pull request #3584: [AIRFLOW-249] Refactor the
SLA mechanism
URL: https://github.com/apache/incubator-airflow/pull/3584#discussion_r224858251
##
File path: airflow/utils/sla.py
##
@@ -0,0 +1,371 @@
+import airflow.models
+from airflow.utils import asciiart
+from airflow.utils.db import provide_session
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.state import State
+
+log = LoggingMixin().log
+
+
+def yield_unscheduled_runs(dag, last_scheduled_run, ts):
+"""
+Yield new DagRuns that haven't been created yet.
+"""
+
+# TODO: A lot of this logic is duplicated from the scheduler. It would
+# be better to have one function that yields upcoming DAG runs in a
+# consistent way that is usable for both use cases.
+
+# Start by assuming that there is no next run.
+next_run_date = None
+
+# The first DAGRun has not been created yet.
+if not last_scheduled_run:
+task_start_dates = [t.start_date for t in dag.tasks]
+if task_start_dates:
+next_run_date = dag.normalize_schedule(min(task_start_dates))
+# The DagRun is @once and has already happened.
+elif dag.schedule_interval == '@once':
+return
+# Start from the next "normal" run.
+else:
+next_run_date =
dag.following_schedule(last_scheduled_run.execution_date)
+
+while True:
+# There should be a next execution.
+if not next_run_date:
+return
+
+# The next execution shouldn't be in the future.
+if next_run_date > ts:
+return
+
+# The next execution shouldn't be beyond the DAG's end date.
+# n.b. - tasks have their own end dates checked later
+if next_run_date and dag.end_date and next_run_date > dag.end_date:
+return
+
+# Calculate the end of this execution period.
+if dag.schedule_interval == '@once':
+period_end = next_run_date
+else:
+period_end = dag.following_schedule(next_run_date)
+
+# The next execution shouldn't still be mid-period.
+if period_end > ts:
+return
+
+# We've passed every filter; this is a valid future DagRun that
+# presumably hasn't been scheduled due to concurrency limits.
+# Create a DAGRun, though it won't exist in the db yet.
+next_run = dag.create_dagrun(
+run_id=airflow.models.DagRun.ID_PREFIX + next_run_date.isoformat(),
+execution_date=next_run_date,
+start_date=ts,
+state=State.RUNNING,
+external_trigger=False
+)
+yield next_run
+
+# Examine the next date.
+next_run_date = dag.following_schedule(next_run_date)
+
+
+def yield_unscheduled_tis(dag_run, ts, session=None):
+"""
+Given an unscheduled `DagRun`, yield any unscheduled TIs that will exist
+for it in the future, respecting the end date of the DAG and task.
+"""
+for task in dag_run.dag.tasks:
+end_dates = []
+if dag_run.dag.end_date:
+end_dates.append(dag_run.dag.end_date)
+if task.end_date:
+end_dates.append(task.end_date)
+
+# Create TIs if there is no end date, or it hasn't happened yet.
+if not end_dates or ts < min(end_dates):
+yield airflow.models.TaskInstance(task, dag_run.execution_date)
+
+
+def get_sla_misses(ti, session):
+"""
+Get all SLA misses that match a particular TaskInstance.
+"""
+SM = airflow.models.SlaMiss
+return session.query(SM).filter(
+SM.dag_id == ti.dag_id,
+SM.task_id == ti.task_id,
+SM.execution_date == ti.execution_date
+).all()
+
+
+def create_sla_misses(ti, ts, session):
+"""
+Determine whether a TaskInstance has missed any SLAs as of a provided
+timestamp. If it has, create `SlaMiss` objects in the provided session.
+Note that one TaskInstance can have multiple SLA miss objects: for example,
+it can both start late and run longer than expected.
+"""
+# Skipped task instances will never trigger SLAs because they
+# were intentionally not scheduled. Though, it's still a valid and
+# interesting SLA miss if a task that's *going* to be skipped today is
+# late! That could mean that an upstream task is hanging.
+if ti.state == State.SKIPPED:
+return
+
+SM = airflow.models.SlaMiss
+
+# Get existing misses.
+ti_misses = {sm.sla_type: sm for sm in get_sla_misses(ti, session)}
+
+# Calculate each type of SLA miss. Wrapping exceptions here is
+# important so that an exception in one type of SLA doesn't
+# prevent other task SLAs from getting triggered.
+
+# SLA Miss for Expected Duration
+if SM.TASK_DURATION_EXCEEDED not in ti_misses \
+and ti.task.expected_duration and ti.start_date:
+try:
+
[GitHub] SamWildmo commented on issue #2879: [AIRFLOW-1921] Added support for presto https and user auth
SamWildmo commented on issue #2879: [AIRFLOW-1921] Added support for presto https and user auth URL: https://github.com/apache/incubator-airflow/pull/2879#issuecomment-429448938 What is the status of this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] SamWildmo commented on issue #3850: [AIRFLOW-3010] added support for date format to postgres_to_gcs_operator
SamWildmo commented on issue #3850: [AIRFLOW-3010] added support for date format to postgres_to_gcs_operator URL: https://github.com/apache/incubator-airflow/pull/3850#issuecomment-429450622 I could use this. Any chance this will be in 1.10.1? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#issuecomment-417764201 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=h1) Report > Merging [#3828](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#3828 +/- ## === Coverage 75.82% 75.82% === Files 199 199 Lines 1594915949 === Hits1209312093 Misses 3856 3856 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=footer). Last update [f43600d...5b9515b](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#issuecomment-417764201 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=h1) Report > Merging [#3828](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#3828 +/- ## === Coverage 75.82% 75.82% === Files 199 199 Lines 1594915949 === Hits1209312093 Misses 3856 3856 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=footer). Last update [f43600d...5b9515b](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant
kaxil commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035#issuecomment-429466365 Awesome... 🎉 Boom This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #4035: [AIRFLOW-3190] Make flake8 compliant
kaxil closed pull request #4035: [AIRFLOW-3190] Make flake8 compliant
URL: https://github.com/apache/incubator-airflow/pull/4035
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.flake8 b/.flake8
new file mode 100644
index 00..2723df1f10
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,3 @@
+[flake8]
+max-line-length = 110
+ignore = E731
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 90452d954b..f7f69ac0b3 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -31,4 +31,4 @@ Make sure you have checked _all_ steps below.
### Code Quality
-- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
+- [ ] Passes `flake8`
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py
b/airflow/contrib/hooks/gcp_dataproc_hook.py
index ca86f08795..e068d65cbe 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -78,8 +78,7 @@ def wait_for_done(self):
def raise_error(self, message=None):
job_state = self.job['status']['state']
# We always consider ERROR to be an error state.
-if ((self.job_error_states and job_state in self.job_error_states)
-or 'ERROR' == job_state):
+if (self.job_error_states and job_state in self.job_error_states) or
'ERROR' == job_state:
ex_message = message or ("Google DataProc job has state: %s" %
job_state)
ex_details = (str(self.job['status']['details'])
if 'details' in self.job['status']
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 3922939a86..0bcb131c72 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -203,10 +203,10 @@ def run(self):
self._execute()
# In case of max runs or max duration
self.state = State.SUCCESS
-except SystemExit as e:
+except SystemExit:
# In case of ^C or SIGTERM
self.state = State.SUCCESS
-except Exception as e:
+except Exception:
self.state = State.FAILED
raise
finally:
@@ -424,7 +424,7 @@ def start(self):
def terminate(self, sigkill=False):
"""
Terminate (and then kill) the process launched to process the file.
-
+
:param sigkill: whether to issue a SIGKILL if SIGTERM doesn't work.
:type sigkill: bool
"""
@@ -453,7 +453,7 @@ def pid(self):
def exit_code(self):
"""
After the process is finished, this can be called to get the return
code
-
+
:return: the exit code of the process
:rtype: int
"""
@@ -465,7 +465,7 @@ def exit_code(self):
def done(self):
"""
Check if the process launched to process this file is done.
-
+
:return: whether the process is finished running
:rtype: bool
"""
@@ -2033,7 +2033,7 @@ def _update_counters(self, ti_status):
"""
Updates the counters per state of the tasks that were running. Can
re-add
to tasks to run in case required.
-
+
:param ti_status: the internal status of the backfill job tasks
:type ti_status: BackfillJob._DagRunTaskStatus
"""
@@ -2078,7 +2078,7 @@ def _manage_executor_state(self, running):
"""
Checks if the executor agrees with the state of task instances
that are running
-
+
:param running: dict of key, task to verify
"""
executor = self.executor
@@ -2110,7 +2110,7 @@ def _get_dag_run(self, run_date, session=None):
Returns a dag run for the given run date, which will be matched to an
existing
dag run if available or create a new dag run otherwise. If the
max_active_runs
limit is reached, this function will return None.
-
+
:param run_date: the execution date for the dag run
:type run_date: datetime
:param session: the database session object
@@ -2170,7 +2170,7 @@ def _task_instances_for_dag_run(self, dag_run,
session=None):
"""
Returns a map of task instance key to task instance object for the
tasks to
run in the given dag run.
-
+
:param dag_run: the dag run to get the tasks from
:type dag_run: models.DagRun
:param session: the database session object
@@ -2236,7 +2236,7 @@ def _process_backfill_task_instances(self,
Process a set of task instances from a set of dag runs. Special
handling is done
to account for different task instance states that could be present
when running
them in a backfill process.
-
[GitHub] kaxil closed pull request #4040: [AIRFLOW-3192] Remove deprecated post_execute logic
kaxil closed pull request #4040: [AIRFLOW-3192] Remove deprecated post_execute
logic
URL: https://github.com/apache/incubator-airflow/pull/4040
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/models.py b/airflow/models.py
index 6154ca64f4..282c2f9e2a 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1650,21 +1650,7 @@ def signal_handler(signum, frame):
if result is not None:
self.xcom_push(key=XCOM_RETURN_KEY, value=result)
-# TODO remove deprecated behavior in Airflow 2.0
-try:
-task_copy.post_execute(context=context, result=result)
-except TypeError as e:
-if 'unexpected keyword argument' in str(e):
-warnings.warn(
-'BaseOperator.post_execute() now takes two '
-'arguments, `context` and `result`, but "{}" only '
-'expected one. This behavior is deprecated and '
-'will be removed in a future version of '
-'Airflow.'.format(self.task_id),
-category=DeprecationWarning)
-task_copy.post_execute(context=context)
-else:
-raise
+task_copy.post_execute(context=context, result=result)
Stats.incr('operator_successes_{}'.format(
self.task.__class__.__name__), 1, 1)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
codecov-io edited a comment on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#issuecomment-427502365 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=h1) Report > Merging [#4008](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4008 +/- ## == + Coverage 75.79% 75.82% +0.03% == Files 199 199 Lines 1594615949 +3 == + Hits1208612094 +8 + Misses 3860 3855 -5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.84% <0%> (-0.42%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `79.31% <0%> (-0.36%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | | [airflow/utils/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jb25maWd1cmF0aW9uLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.71% <0%> (ø)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.18% <0%> (+0.14%)` | :arrow_up: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `88% <0%> (+0.32%)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `65.01% <0%> (+0.53%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=footer). Last update [b8be322...ce9257f](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
codecov-io edited a comment on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#issuecomment-427502365 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=h1) Report > Merging [#4008](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4008 +/- ## == + Coverage 75.79% 75.82% +0.03% == Files 199 199 Lines 1594615949 +3 == + Hits1208612094 +8 + Misses 3860 3855 -5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.84% <0%> (-0.42%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `79.31% <0%> (-0.36%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | | [airflow/utils/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jb25maWd1cmF0aW9uLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.71% <0%> (ø)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.18% <0%> (+0.14%)` | :arrow_up: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `88% <0%> (+0.32%)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4008/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `65.01% <0%> (+0.53%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=footer). Last update [b8be322...ce9257f](https://codecov.io/gh/apache/incubator-airflow/pull/4008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
kaxil commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file URL: https://github.com/apache/incubator-airflow/pull/3893#issuecomment-429470293 I have made the changes. Let's wait for the CI to pass and it should be good to be merged :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
kaxil commented on a change in pull request #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#discussion_r224923649 ## File path: airflow/contrib/operators/gcs_to_gcs.py ## @@ -62,6 +63,11 @@ class GoogleCloudStorageToGoogleCloudStorageOperator(BaseOperator): For this to work, the service account making the request must have domain-wide delegation enabled. :type delegate_to: str +:param filter_by_last_modified: When True, only copies/moves source object(s) +that were last modified after last_modified_time +:type: bool Review comment: This is not the correct way to mention datatype. It should be ``` :type filter_by_last_modified: bool ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] neil90 commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
neil90 commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file URL: https://github.com/apache/incubator-airflow/pull/3893#issuecomment-429472104 Ohh thanks @kaxil sorry about that I was gonna try to get to today after work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
kaxil commented on a change in pull request #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#discussion_r224923730 ## File path: airflow/contrib/operators/gcs_to_gcs.py ## @@ -62,6 +63,11 @@ class GoogleCloudStorageToGoogleCloudStorageOperator(BaseOperator): For this to work, the service account making the request must have domain-wide delegation enabled. :type delegate_to: str +:param filter_by_last_modified: When True, only copies/moves source object(s) +that were last modified after last_modified_time +:type: bool +:param last_modified_time: Timestamp in GMT to filter source object(s) +:type datetime Review comment: Same here. ``` :type last_modified_time: datetime ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
codecov-io edited a comment on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038#issuecomment-429203577 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=h1) Report > Merging [#4038](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `63.63%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4038 +/- ## == + Coverage 75.82% 75.83% +<.01% == Files 199 199 Lines 1594915950 +1 == + Hits1209312095 +2 + Misses 3856 3855 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.41% <50%> (+0.56%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.65% <66.66%> (-0.03%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=footer). Last update [f43600d...d2e5bed](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
codecov-io edited a comment on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038#issuecomment-429203577 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=h1) Report > Merging [#4038](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `63.63%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4038 +/- ## == + Coverage 75.82% 75.83% +<.01% == Files 199 199 Lines 1594915950 +1 == + Hits1209312095 +2 + Misses 3856 3855 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.41% <50%> (+0.56%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.65% <66.66%> (-0.03%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=footer). Last update [f43600d...d2e5bed](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API
codecov-io commented on issue #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API URL: https://github.com/apache/incubator-airflow/pull/4044#issuecomment-429484705 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=h1) Report > Merging [#4044](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4044 +/- ## == + Coverage 75.82% 75.82% +<.01% == Files 199 199 Lines 1594915949 == + Hits1209312094 +1 + Misses 3856 3855 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4044/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.6% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4044/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.71% <0%> (+0.04%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=footer). Last update [f43600d...16c14ef](https://codecov.io/gh/apache/incubator-airflow/pull/4044?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#issuecomment-417764201 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=h1) Report > Merging [#3828](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f43600db1efde4d924f50e51127cf522de79e68e?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3828 +/- ## == + Coverage 75.82% 75.82% +<.01% == Files 199 199 Lines 1594915949 == + Hits1209312094 +1 + Misses 3856 3855 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3828/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.71% <0%> (+0.04%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=footer). Last update [f43600d...5b9515b](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sid88in opened a new pull request #4045: [AIRFLOW-3201]
sid88in opened a new pull request #4045: [AIRFLOW-3201]
URL: https://github.com/apache/incubator-airflow/pull/4045
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chandulal opened a new pull request #4046: Revert: Need some approval to add name in companies list.
chandulal opened a new pull request #4046: Revert: Need some approval to add name in companies list. URL: https://github.com/apache/incubator-airflow/pull/4046 ### Description Need approval to add name in companies list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sid88in commented on issue #4045: [AIRFLOW-3201] Update readme
sid88in commented on issue #4045: [AIRFLOW-3201] Update readme URL: https://github.com/apache/incubator-airflow/pull/4045#issuecomment-429493851 https://user-images.githubusercontent.com/1587005/46898828-838c3480-ce41-11e8-9f34-710a26150ba6.png";> doesn't look like the test failed because adding my name in readme :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp opened a new pull request #4047: [AIRFLOW-3193] Handle recent docker-py versions.
jmcarp opened a new pull request #4047: [AIRFLOW-3193] Handle recent docker-py
versions.
URL: https://github.com/apache/incubator-airflow/pull/4047
Note: this is an alternative to #4042. Does this make sense @deagon?
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Prior to docker-py 3.0, cpu and memory limits could be passed to either
`create_container` or `create_host_config`. After version 3.0, those
options can only be passed to `create_host_config`. To support current
versions of docker-py, pass resource limit options to
`create_host_config`.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sid88in opened a new pull request #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator
sid88in opened a new pull request #4048: [AIRFLOW-3202] add missing
documentation for AWS hooks/operator
URL: https://github.com/apache/incubator-airflow/pull/4048
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sid88in commented on issue #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator
sid88in commented on issue #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator URL: https://github.com/apache/incubator-airflow/pull/4048#issuecomment-429512934 Sometime back I added AWS hooks/operators but don't see them in the documentation, so adding them now :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4045: [AIRFLOW-3201] Update readme
Fokko commented on issue #4045: [AIRFLOW-3201] Update readme URL: https://github.com/apache/incubator-airflow/pull/4045#issuecomment-429513084 The CI is definitely unrelated. Welcome aboard @sid88in ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #4045: [AIRFLOW-3201] Update readme
Fokko closed pull request #4045: [AIRFLOW-3201] Update readme URL: https://github.com/apache/incubator-airflow/pull/4045 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index 0f0d6c79ff..9560514576 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,7 @@ Currently **officially** using Airflow: 1. [G Adventures](https://gadventures.com) [[@samuelmullin](https://github.com/samuelmullin)] 1. [GameWisp](https://gamewisp.com) [[@tjbiii](https://github.com/TJBIII) & [@theryanwalls](https://github.com/theryanwalls)] 1. [Gentner Lab](http://github.com/gentnerlab) [[@neuromusic](https://github.com/neuromusic)] -1. [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk)] +1. [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk) & [@sid88in](https://github.com/sid88in)] 1. [Global Fashion Group](http://global-fashion-group.com) [[@GFG](https://github.com/GFG)] 1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng) & [@datagovsg](https://github.com/datagovsg)] 1. [Grab](https://www.grab.com/sg/) [[@grab](https://github.com/grab)] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4046: Revert: Need some approval to add name in companies list.
Fokko commented on issue #4046: Revert: Need some approval to add name in companies list. URL: https://github.com/apache/incubator-airflow/pull/4046#issuecomment-429513222 @chandulal The changes are empty, please open a new PR with the reverted commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator
Fokko commented on a change in pull request #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator URL: https://github.com/apache/incubator-airflow/pull/4048#discussion_r224950667 ## File path: docs/integration.rst ## @@ -313,6 +313,49 @@ S3ToRedshiftTransfer .. autoclass:: airflow.operators.s3_to_redshift_operator.S3ToRedshiftTransfer +AWS DynamoDB + Review comment: Can you align the ``: ``` AWS DynamoDB ``` Otherwise this will generate warnings at compile time. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
Fokko commented on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#issuecomment-429514162 @bkvarda I have one more thought, let me know what you think of it. Right now we have two arguments added to the public API. `filter_by_last_modified` and `last_modified_time`, where `last_modified_time` is set to the current datetime. What if we make `last_modified_time` optional, and if you set it, then it will be used. This will simplify the usage of the operator, since you don't need to set `filter_by_last_modified` as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
Fokko commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file URL: https://github.com/apache/incubator-airflow/pull/3893#issuecomment-429514276 Thanks @neil90 @kaxil. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
Fokko closed pull request #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow
compression of file
URL: https://github.com/apache/incubator-airflow/pull/3893
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/gcs_hook.py
b/airflow/contrib/hooks/gcs_hook.py
index 6cfa1cf565..ca4b2a7754 100644
--- a/airflow/contrib/hooks/gcs_hook.py
+++ b/airflow/contrib/hooks/gcs_hook.py
@@ -24,7 +24,10 @@
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.exceptions import AirflowException
+import gzip as gz
+import shutil
import re
+import os
class GoogleCloudStorageHook(GoogleCloudBaseHook):
@@ -172,7 +175,8 @@ def download(self, bucket, object, filename=None):
return downloaded_file_bytes
# pylint:disable=redefined-builtin
-def upload(self, bucket, object, filename,
mime_type='application/octet-stream'):
+def upload(self, bucket, object, filename,
+ mime_type='application/octet-stream', gzip=False):
"""
Uploads a local file to Google Cloud Storage.
@@ -184,14 +188,30 @@ def upload(self, bucket, object, filename,
mime_type='application/octet-stream')
:type filename: str
:param mime_type: The MIME type to set when uploading the file.
:type mime_type: str
+:param gzip: Option to compress file for upload
+:type gzip: bool
"""
service = self.get_conn()
+
+if gzip:
+filename_gz = filename + '.gz'
+
+with open(filename, 'rb') as f_in:
+with gz.open(filename_gz, 'wb') as f_out:
+shutil.copyfileobj(f_in, f_out)
+filename = filename_gz
+
media = MediaFileUpload(filename, mime_type)
+
try:
service \
.objects() \
.insert(bucket=bucket, name=object, media_body=media) \
.execute()
+
+# Clean up gzip file
+if gzip:
+os.remove(filename)
return True
except errors.HttpError as ex:
if ex.resp['status'] == '404':
diff --git a/airflow/contrib/operators/file_to_gcs.py
b/airflow/contrib/operators/file_to_gcs.py
index a392a16891..de40abd473 100644
--- a/airflow/contrib/operators/file_to_gcs.py
+++ b/airflow/contrib/operators/file_to_gcs.py
@@ -25,7 +25,8 @@
class FileToGoogleCloudStorageOperator(BaseOperator):
"""
-Uploads a file to Google Cloud Storage
+Uploads a file to Google Cloud Storage.
+Optionally can compress the file for upload.
:param src: Path to the local file. (templated)
:type src: str
@@ -39,6 +40,8 @@ class FileToGoogleCloudStorageOperator(BaseOperator):
:type mime_type: str
:param delegate_to: The account to impersonate, if any
:type delegate_to: str
+:param gzip: Allows for file to be compressed and uploaded as gzip
+:type gzip: bool
"""
template_fields = ('src', 'dst', 'bucket')
@@ -50,6 +53,7 @@ def __init__(self,
google_cloud_storage_conn_id='google_cloud_default',
mime_type='application/octet-stream',
delegate_to=None,
+ gzip=False,
*args,
**kwargs):
super(FileToGoogleCloudStorageOperator, self).__init__(*args, **kwargs)
@@ -59,6 +63,7 @@ def __init__(self,
self.google_cloud_storage_conn_id = google_cloud_storage_conn_id
self.mime_type = mime_type
self.delegate_to = delegate_to
+self.gzip = gzip
def execute(self, context):
"""
@@ -72,4 +77,6 @@ def execute(self, context):
bucket=self.bucket,
object=self.dst,
mime_type=self.mime_type,
-filename=self.src)
+filename=self.src,
+gzip=self.gzip,
+)
diff --git a/tests/contrib/operators/test_file_to_gcs.py
b/tests/contrib/operators/test_file_to_gcs.py
new file mode 100644
index 00..590267a319
--- /dev/null
+++ b/tests/contrib/operators/test_file_to_gcs.py
@@ -0,0 +1,66 @@
+import datetime
+import unittest
+
+from airflow import DAG, configuration
+from airflow.contrib.operators.file_to_gcs import
FileToGoogleCloudStorageOperator
+
+try:
+from unittest import mock
+except ImportError:
+try:
+import mock
+except ImportError:
+mock = None
+
+
+class TestFileToGcsOperator(unittest.TestCase):
+
+_config = {
+'src': '/tmp/fake.csv',
+'dst': 'fake.csv',
+'bucket': 'dummy',
+'mime_type': 'application/octet-stream',
+'gzip': False
+}
+
+def setUp(self):
+configuration.load_test_config()
+args = {
+'ow
[GitHub] Fokko closed pull request #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks
Fokko closed pull request #4043: [AIRFLOW-3174] Refine Docstring for SQL
Operators & Hooks
URL: https://github.com/apache/incubator-airflow/pull/4043
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index 5b50ade34e..0bcbdf2f11 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -191,6 +191,7 @@ def get_autocommit(self, conn):
Return True if conn.autocommit is set to True.
Return False if conn.autocommit is not set or set to False or conn
does not support autocommit.
+
:param conn: Connection to get autocommit setting from.
:type conn: connection object.
:return: connection autocommit setting.
diff --git a/airflow/hooks/mysql_hook.py b/airflow/hooks/mysql_hook.py
index 51b53c1f8f..9ec8ca4d3e 100644
--- a/airflow/hooks/mysql_hook.py
+++ b/airflow/hooks/mysql_hook.py
@@ -49,6 +49,7 @@ def set_autocommit(self, conn, autocommit):
def get_autocommit(self, conn):
"""
MySql connection gets autocommit in a different way.
+
:param conn: connection to get autocommit setting from.
:type conn: connection object.
:return: connection autocommit setting
diff --git a/airflow/operators/jdbc_operator.py
b/airflow/operators/jdbc_operator.py
index a10bf30c5c..1c9423d5ab 100644
--- a/airflow/operators/jdbc_operator.py
+++ b/airflow/operators/jdbc_operator.py
@@ -28,12 +28,17 @@ class JdbcOperator(BaseOperator):
Requires jaydebeapi.
-:param jdbc_conn_id: reference to a predefined database
-:type jdbc_conn_id: str
:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,
a list of str (sql statements), or reference to a template file.
Template reference are recognized by str ending in '.sql'
+:param jdbc_conn_id: reference to a predefined database
+:type jdbc_conn_id: str
+:param autocommit: if True, each command is automatically committed.
+(default value: False)
+:type autocommit: bool
+:param parameters: (optional) the parameters to render the SQL query with.
+:type parameters: mapping or iterable
"""
template_fields = ('sql',)
diff --git a/airflow/operators/mssql_operator.py
b/airflow/operators/mssql_operator.py
index f27e2145eb..086d1e39a9 100644
--- a/airflow/operators/mssql_operator.py
+++ b/airflow/operators/mssql_operator.py
@@ -26,11 +26,16 @@ class MsSqlOperator(BaseOperator):
"""
Executes sql code in a specific Microsoft SQL database
-:param mssql_conn_id: reference to a specific mssql database
-:type mssql_conn_id: str
:param sql: the sql code to be executed
:type sql: str or string pointing to a template file with .sql
extension. (templated)
+:param mssql_conn_id: reference to a specific mssql database
+:type mssql_conn_id: str
+:param parameters: (optional) the parameters to render the SQL query with.
+:type parameters: mapping or iterable
+:param autocommit: if True, each command is automatically committed.
+(default value: False)
+:type autocommit: bool
:param database: name of database which overwrite defined one in connection
:type database: str
"""
diff --git a/airflow/operators/mysql_operator.py
b/airflow/operators/mysql_operator.py
index e5bd788bd1..2cc9b07e34 100644
--- a/airflow/operators/mysql_operator.py
+++ b/airflow/operators/mysql_operator.py
@@ -26,12 +26,17 @@ class MySqlOperator(BaseOperator):
"""
Executes sql code in a specific MySQL database
-:param mysql_conn_id: reference to a specific mysql database
-:type mysql_conn_id: str
:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,
a list of str (sql statements), or reference to a template file.
Template reference are recognized by str ending in '.sql'
+:param mysql_conn_id: reference to a specific mysql database
+:type mysql_conn_id: str
+:param parameters: (optional) the parameters to render the SQL query with.
+:type parameters: mapping or iterable
+:param autocommit: if True, each command is automatically committed.
+(default value: False)
+:type autocommit: bool
:param database: name of database which overwrite defined one in connection
:type database: str
"""
diff --git a/airflow/operators/oracle_operator.py
b/airflow/operators/oracle_operator.py
index b97129715f..2d1a655f9a 100644
--- a/airflow/operators/oracle_operator.py
+++ b/airflow/operators/oracle_operator.py
@@ -26,12 +26,17 @@ class OracleOperator(BaseOperator):
"""
Execu
[GitHub] XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks
XD-DENG commented on issue #4043: [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks URL: https://github.com/apache/incubator-airflow/pull/4043#issuecomment-429514750 Thanks @Fokko for merging :-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table
Fokko commented on issue #4003: [AIRFLOW-3163] add operator to enable setting
table description in BigQuery table
URL:
https://github.com/apache/incubator-airflow/pull/4003#issuecomment-429515036
Just thinking along here. We could extend the create table definition and
add the description over there:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L214-L284
The create table call supports supplying a description field:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
We could template this field, so we can use `created at: {{ ds }}`. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko edited a comment on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table
Fokko edited a comment on issue #4003: [AIRFLOW-3163] add operator to enable
setting table description in BigQuery table
URL:
https://github.com/apache/incubator-airflow/pull/4003#issuecomment-429515036
Just thinking along here. We could extend the create table definition and
add the description over there:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L214-L284
The create table call supports supplying a description field:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
We could template this field, so we can use `created at: {{ execution_date
}}`, which will translate to `created at: 2018-06-04`. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API
Fokko commented on a change in pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API URL: https://github.com/apache/incubator-airflow/pull/4044#discussion_r224951208 ## File path: UPDATING.md ## @@ -72,6 +72,16 @@ then you need to change it like this def is_active(self): return self.active +### EMRHook now passes all of connection's extra to CreateJobFlow API + +EMRHook.create_job_flow has been changed to pass all keys to the create_job_flow API, rather than +just specific known keys for greater flexibility. Review comment: Not only greater flexibility, but less error prone :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API
Fokko commented on a change in pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API URL: https://github.com/apache/incubator-airflow/pull/4044#discussion_r224951208 ## File path: UPDATING.md ## @@ -72,6 +72,16 @@ then you need to change it like this def is_active(self): return self.active +### EMRHook now passes all of connection's extra to CreateJobFlow API + +EMRHook.create_job_flow has been changed to pass all keys to the create_job_flow API, rather than +just specific known keys for greater flexibility. Review comment: Not only greater flexibility, but also less error prone :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API
Fokko closed pull request #4044: [AIRFLOW-3197] EMRHook is missing new parameters of the AWS API URL: https://github.com/apache/incubator-airflow/pull/4044 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator
Fokko closed pull request #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator URL: https://github.com/apache/incubator-airflow/pull/4048 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/code.rst b/docs/code.rst index 764d6949ac..211e1abafe 100644 --- a/docs/code.rst +++ b/docs/code.rst @@ -378,6 +378,7 @@ Community contributed hooks .. autoclass:: airflow.contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook .. autoclass:: airflow.contrib.hooks.aws_hook.AwsHook .. autoclass:: airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook +.. autoclass:: airflow.contrib.hooks.aws_firehose_hook.AwsFirehoseHook .. autoclass:: airflow.contrib.hooks.azure_data_lake_hook.AzureDataLakeHook .. autoclass:: airflow.contrib.hooks.azure_fileshare_hook.AzureFileShareHook .. autoclass:: airflow.contrib.hooks.bigquery_hook.BigQueryHook diff --git a/docs/integration.rst b/docs/integration.rst index 3a5a3c3e05..a7e21be20c 100644 --- a/docs/integration.rst +++ b/docs/integration.rst @@ -313,6 +313,49 @@ S3ToRedshiftTransfer .. autoclass:: airflow.operators.s3_to_redshift_operator.S3ToRedshiftTransfer +AWS DynamoDB + + +- :ref:`HiveToDynamoDBTransferOperator` : Moves data from Hive to DynamoDB. +- :ref:`AwsDynamoDBHook` : Interact with AWS DynamoDB. + +.. _HiveToDynamoDBTransferOperator: + +HiveToDynamoDBTransferOperator +"" + +.. autoclass:: airflow.contrib.operators.hive_to_dynamodb.HiveToDynamoDBTransferOperator + +.. _AwsDynamoDBHook: + +AwsDynamoDBHook +""" + +.. autoclass:: airflow.contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook + +AWS Lambda +'' + +- :ref:`AwsLambdaHook` : Interact with AWS Lambda. + +.. _AwsLambdaHook: + +AwsLambdaHook +" + +.. autoclass:: airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook + +AWS Kinesis +''' + +- :ref:`AwsFirehoseHook` : Interact with AWS Kinesis Firehose. + +.. _AwsFirehoseHook: + +AwsFirehoseHook +""" + +.. autoclass:: airflow.contrib.hooks.aws_firehose_hook.AwsFirehoseHook .. _Databricks: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sid88in commented on issue #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator
sid88in commented on issue #4048: [AIRFLOW-3202] add missing documentation for AWS hooks/operator URL: https://github.com/apache/incubator-airflow/pull/4048#issuecomment-429518493 thanks @Fokko !! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG opened a new pull request #4049: [AIRFLOW-3203] Fix DockerOperator & its test
XD-DENG opened a new pull request #4049: [AIRFLOW-3203] Fix DockerOperator & its test URL: https://github.com/apache/incubator-airflow/pull/4049 ### Jira - https://issues.apache.org/jira/browse/AIRFLOW-3203 ### Description - For argument `image`, no need to explicitly add "latest" if tag is omitted. "latest" will be used by default if no tag provided. This is handled by `docker` package itself. - Intermediate variable `cpu_shares` is not needed. - Fix wrong usage of `cpu_shares` and `cpu_shares`. Based on https://docker-py.readthedocs.io/en/stable/api.html#docker.api.container.ContainerApiMixin.create_host_config, They should be an arguments of self.cli.create_host_config() rather than APIClient.create_container(). - Change name of the corresponding test script, to ensure it can be discovered. - Fix the test itself. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chandulal commented on issue #4046: Revert: Need some approval to add name in companies list.
chandulal commented on issue #4046: Revert: Need some approval to add name in companies list. URL: https://github.com/apache/incubator-airflow/pull/4046#issuecomment-429552932 @Fokko There is only one change in README.md. Just removed "Grab" from company list. Because we need to take permission from the legal team. These changes are there. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test
codecov-io commented on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test URL: https://github.com/apache/incubator-airflow/pull/4049#issuecomment-429554020 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=h1) Report > Merging [#4049](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b862ca8e1fb1a6c5f75a9f04b7c323853b2ae152?src=pr&el=desc) will **increase** coverage by `0.56%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4049 +/- ## == + Coverage 75.87% 76.43% +0.56% == Files 199 199 Lines 1594315939 -4 == + Hits1209612183 +87 + Misses 3847 3756 -91 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `97.61% <100%> (+97.61%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.39% <0%> (+0.26%)` | :arrow_up: | | [airflow/utils/file.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9maWxlLnB5) | `84% <0%> (+8%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=footer). Last update [b862ca8...3972467](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test
codecov-io edited a comment on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test URL: https://github.com/apache/incubator-airflow/pull/4049#issuecomment-429554020 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=h1) Report > Merging [#4049](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b862ca8e1fb1a6c5f75a9f04b7c323853b2ae152?src=pr&el=desc) will **increase** coverage by `0.56%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4049 +/- ## == + Coverage 75.87% 76.43% +0.56% == Files 199 199 Lines 1594315939 -4 == + Hits1209612183 +87 + Misses 3847 3756 -91 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `97.61% <100%> (+97.61%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.39% <0%> (+0.26%)` | :arrow_up: | | [airflow/utils/file.py](https://codecov.io/gh/apache/incubator-airflow/pull/4049/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9maWxlLnB5) | `84% <0%> (+8%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=footer). Last update [b862ca8...3972467](https://codecov.io/gh/apache/incubator-airflow/pull/4049?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3914: [AIRFLOW-3069] Log all output of the S3 file transform script
codecov-io edited a comment on issue #3914: [AIRFLOW-3069] Log all output of the S3 file transform script URL: https://github.com/apache/incubator-airflow/pull/3914#issuecomment-422548293 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=h1) Report > Merging [#3914](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b862ca8e1fb1a6c5f75a9f04b7c323853b2ae152?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3914 +/- ## == + Coverage 75.87% 75.88% +0.01% == Files 199 199 Lines 1594315947 +4 == + Hits1209612101 +5 + Misses 3847 3846 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/s3\_file\_transform\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3914/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfZmlsZV90cmFuc2Zvcm1fb3BlcmF0b3IucHk=) | `96.22% <100%> (+2.34%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=footer). Last update [b862ca8...ea576d2](https://codecov.io/gh/apache/incubator-airflow/pull/3914?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sbilinski commented on issue #3914: [AIRFLOW-3069] Log all output of the S3 file transform script
sbilinski commented on issue #3914: [AIRFLOW-3069] Log all output of the S3 file transform script URL: https://github.com/apache/incubator-airflow/pull/3914#issuecomment-429556983 @Fokko I've applied some changes and rebased against `master` - please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bkvarda commented on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator
bkvarda commented on issue #4008: [AIRFLOW-3155] Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator URL: https://github.com/apache/incubator-airflow/pull/4008#issuecomment-429572002 @Fokko I agree - I’ll make that change This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
codecov-io edited a comment on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-409915837 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=h1) Report > Merging [#3683](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr&el=desc) will **increase** coverage by `1.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#3683 +/- ## == + Coverage 75.79% 77.53% +1.74% == Files 199 205 +6 Lines 1594615771 -175 == + Hits1208612228 +142 + Misses 3860 3543 -317 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `31.03% <0%> (-68.97%)` | :arrow_down: | | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `41.17% <0%> (-58.83%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `67.07% <0%> (-17.31%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `78% <0%> (-12%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `73.91% <0%> (-7.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `83.95% <0%> (-5.31%)` | :arrow_down: | | [airflow/utils/state.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `93.33% <0%> (-3.34%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.58% <0%> (-3.13%)` | :arrow_down: | | [airflow/www\_rbac/utils.py](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy91dGlscy5weQ==) | `66.21% <0%> (-2.73%)` | :arrow_down: | | ... and [74 more](https://codecov.io/gh/apache/incubator-airflow/pull/3683/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=footer). Last update [b8be322...9401914](https://codecov.io/gh/apache/incubator-airflow/pull/3683?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test
XD-DENG commented on issue #4049: [AIRFLOW-3203] Fix DockerOperator & its test URL: https://github.com/apache/incubator-airflow/pull/4049#issuecomment-429591267 Hi @Fokko, As we discussed in the email list yesterday, some scripts in `/test` should be renamed to ensure they can be discovered by the CI process. This PR only addresses issues for `DockerOperator` though. There are some issues in both the `DockerOperator` itself & its test (including name of the test script). For other operators whose test were missed due to file name, I suspect there are accumulated issues to fix as well (it may not be as easy as just prepending `test_` to the test file name). I may take a look into them later when I get time. Cheers This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2450: [Airflow-1413] Fix FTPSensor failing on error message with unexpected text.
ron819 commented on issue #2450: [Airflow-1413] Fix FTPSensor failing on error message with unexpected text. URL: https://github.com/apache/incubator-airflow/pull/2450#issuecomment-429600699 @Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429043686 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=h1) Report > Merging [#4036](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b862ca8e1fb1a6c5f75a9f04b7c323853b2ae152?src=pr&el=desc) will **increase** coverage by `0.02%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4036 +/- ## == + Coverage 75.87% 75.89% +0.02% == Files 199 199 Lines 1594315960 +17 == + Hits1209612113 +17 Misses 3847 3847 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `91.95% <100%> (+0.59%)` | :arrow_up: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `97.05% <100%> (+0.35%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=footer). Last update [b862ca8...9e7b1b8](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429043686 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=h1) Report > Merging [#4036](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b862ca8e1fb1a6c5f75a9f04b7c323853b2ae152?src=pr&el=desc) will **increase** coverage by `0.02%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4036 +/- ## == + Coverage 75.87% 75.89% +0.02% == Files 199 199 Lines 1594315960 +17 == + Hits1209612113 +17 Misses 3847 3847 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `91.95% <100%> (+0.59%)` | :arrow_up: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `97.05% <100%> (+0.35%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=footer). Last update [b862ca8...9e7b1b8](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] antxxxx commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table
ant commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table URL: https://github.com/apache/incubator-airflow/pull/4003#issuecomment-429617980 @Fokko I am not explicitly creating a table - I am either loading into a new table or sending query results to a new table. With these methods you can not set the table description (this is a limitation of bigquery api) @tswast Having an operator to update table metadata is a good idea. I think the only things you can actually update in a patch call are description, labels and expiration. Would you expect these as 3 specific parameters to the operator, or one parameter whose value is passed to the patch call? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb opened a new pull request #4050: [AIRFLOW-3178] Don't bake ENV and _cmd into tmp config for non-sudo
ashb opened a new pull request #4050: [AIRFLOW-3178] Don't bake ENV and _cmd
into tmp config for non-sudo
URL: https://github.com/apache/incubator-airflow/pull/4050
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Further to #4029,
If we are running tasks via sudo then AIRFLOW__ config env vars won't be
visible anymore (without them showing up in `ps`) and we likely might
not have permission to run the _cmd's specified to find the passwords.
But if we are running as the same user then there is no need to "bake"
those options in to the temporary config file -- if the operator decided
they didn't want those values appearing in a config file on disk, then
lets do our best to respect that.
This also removes a possible race condition that would make temporary
config file be readable by more than the airflow or run-as user
The exact behaviour would depend on the umask we run under, and the
primary group of our user, likely this would mean the file was readably
by members of the airflow group (which in most cases would be just the
airflow user). To remove any such possibility set the umask before we
write anything to the file.
### Tests
- [x] My PR adds the following unit tests: added some tests to
tests/configuration.py.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
