[GitHub] [airflow] codecov-io edited a comment on issue #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI
codecov-io edited a comment on issue #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI URL: https://github.com/apache/airflow/pull/7240#issuecomment-577059183 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=h1) Report > Merging [#7240](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr&el=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7240 +/- ## == - Coverage 85.34% 85.06% -0.28% == Files 791 791 Lines 4012440124 == - Hits3424334131 -112 - Misses 5881 5993 +112 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.28% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=footer). Last update [21a4651...ff867b7](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI
codecov-io commented on issue #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI URL: https://github.com/apache/airflow/pull/7240#issuecomment-577059183 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=h1) Report > Merging [#7240](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr&el=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#7240 +/- ## == - Coverage 85.34% 85.06% -0.28% == Files 791 791 Lines 4012440124 == - Hits3424334131 -112 - Misses 5881 5993 +112 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.28% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7240/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=footer). Last update [21a4651...ff867b7](https://codecov.io/gh/apache/airflow/pull/7240?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KevinYang21 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end_of_log mark log record
KevinYang21 commented on a change in pull request #7141: [AIRFLOW-6544] add
log_id to end_of_log mark log record
URL: https://github.com/apache/airflow/pull/7141#discussion_r369425422
##
File path: airflow/utils/log/es_task_handler.py
##
@@ -255,7 +256,9 @@ def close(self):
# Mark the end of file using end of log mark,
# so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
Review comment:
sry I was get lost in my inbox, yes it is very clear to me now. Thank you
for filling me in with details!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] KevinYang21 commented on a change in pull request #7141: [AIRFLOW-6544] add log_id to end_of_log mark log record
KevinYang21 commented on a change in pull request #7141: [AIRFLOW-6544] add
log_id to end_of_log mark log record
URL: https://github.com/apache/airflow/pull/7141#discussion_r369425422
##
File path: airflow/utils/log/es_task_handler.py
##
@@ -255,7 +256,9 @@ def close(self):
# Mark the end of file using end of log mark,
# so we know where to stop while auto-tailing.
-self.handler.stream.write(self.end_of_log_mark)
+if self.write_stdout:
+print()
+self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
Review comment:
sry I was get lost in my inbox, yes it is very clear to me now. Thank you
for filling me in with details!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] yuqian90 commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
yuqian90 commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139#issuecomment-577077316 Some new test failures appeared after I git rebased onto `upstream/v1-10-test`. None of them look related to my change. @potiuk might have better ideas regarding those failures since I see he's committing changes to fix some of them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
codecov-io edited a comment on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#issuecomment-571189190 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=h1) Report > Merging [#6824](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/be812bd660fac4621a25f3269fded8d4e03b0023?src=pr&el=desc) will **decrease** coverage by `0.24%`. > The diff coverage is `97.24%`. [](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6824 +/- ## = - Coverage 84.85% 84.6% -0.25% = Files 679 794 +115 Lines 38536 40233+1697 = + Hits32698 34038+1340 - Misses 58386195 +357 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...ow/providers/amazon/aws/sensors/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL2Nsb3VkX2Zvcm1hdGlvbi5weQ==) | `100% <100%> (ø)` | | | [.../providers/amazon/aws/operators/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvY2xvdWRfZm9ybWF0aW9uLnB5) | `95.34% <95.34%> (ø)` | | | [...flow/providers/amazon/aws/hooks/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9jbG91ZF9mb3JtYXRpb24ucHk=) | `96.87% <96.87%> (ø)` | | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...rflow/contrib/sensors/sagemaker\_training\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYWluaW5nX3NlbnNvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | ... and [390 more](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=footer). Last update [be812bd...3dce509](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
codecov-io edited a comment on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#issuecomment-571189190 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=h1) Report > Merging [#6824](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/be812bd660fac4621a25f3269fded8d4e03b0023?src=pr&el=desc) will **decrease** coverage by `0.24%`. > The diff coverage is `97.24%`. [](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6824 +/- ## = - Coverage 84.85% 84.6% -0.25% = Files 679 794 +115 Lines 38536 40233+1697 = + Hits32698 34038+1340 - Misses 58386195 +357 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...ow/providers/amazon/aws/sensors/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL2Nsb3VkX2Zvcm1hdGlvbi5weQ==) | `100% <100%> (ø)` | | | [.../providers/amazon/aws/operators/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvY2xvdWRfZm9ybWF0aW9uLnB5) | `95.34% <95.34%> (ø)` | | | [...flow/providers/amazon/aws/hooks/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9jbG91ZF9mb3JtYXRpb24ucHk=) | `96.87% <96.87%> (ø)` | | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...rflow/contrib/sensors/sagemaker\_training\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYWluaW5nX3NlbnNvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | ... and [390 more](https://codecov.io/gh/apache/airflow/pull/6824/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=footer). Last update [be812bd...3dce509](https://codecov.io/gh/apache/airflow/pull/6824?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI
kaxil merged pull request #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI URL: https://github.com/apache/airflow/pull/7240 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6615) Remove double sorted in task_list CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-6615. - Fix Version/s: 2.0.0 Resolution: Fixed > Remove double sorted in task_list CLI > - > > Key: AIRFLOW-6615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6615 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.7 >Reporter: Xinbin Huang >Assignee: Xinbin Huang >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6615) Remove double sorted in task_list CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020925#comment-17020925 ] ASF subversion and git services commented on AIRFLOW-6615: -- Commit 1fa7ac1b3978469ba3ec9b902b4f2b4890f8199d in airflow's branch refs/heads/master from Xinbin Huang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1fa7ac1 ] [AIRFLOW-6615] Remove double sorted in task_list CLI (#7240) > Remove double sorted in task_list CLI > - > > Key: AIRFLOW-6615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6615 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.7 >Reporter: Xinbin Huang >Assignee: Xinbin Huang >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6615) Remove double sorted in task_list CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020924#comment-17020924 ] ASF GitHub Bot commented on AIRFLOW-6615: - kaxil commented on pull request #7240: [AIRFLOW-6615] Remove double sorted in task_list CLI URL: https://github.com/apache/airflow/pull/7240 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove double sorted in task_list CLI > - > > Key: AIRFLOW-6615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6615 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.7 >Reporter: Xinbin Huang >Assignee: Xinbin Huang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6615) Remove double sorted in task_list CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6615: Description: tasks list is sorted twice in task_list CLI. > Remove double sorted in task_list CLI > - > > Key: AIRFLOW-6615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6615 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.10.7 >Reporter: Xinbin Huang >Assignee: Xinbin Huang >Priority: Major > Fix For: 2.0.0 > > > tasks list is sorted twice in task_list CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
feluelle commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#issuecomment-577105396 @m1racoli do you think we should add a note to the Updating.md so that people are aware that now the trigger rules will be checked correctly/differently ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
feluelle commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#discussion_r369467378 ## File path: docs/concepts.rst ## @@ -1078,15 +1078,14 @@ For example, consider the following DAG: trigger_rule=TriggerRule.ALL_DONE) task4.set_upstream([task1, task2]) -In the case of this DAG, the ``latest_only`` task will show up as skipped -for all runs except the latest run. ``task1`` is directly downstream of -``latest_only`` and will also skip for all runs except the latest. +In the case of this DAG, the task ``task1`` is directly downstream of +``latest_only`` and will be skipped for all runs except the latest. ``task2`` is entirely independent of ``latest_only`` and will run in all scheduled periods. ``task3`` is downstream of ``task1`` and ``task2`` and because of the default ``trigger_rule`` being ``all_success`` will receive a cascaded skip from ``task1``. ``task4`` is downstream of ``task1`` and -``task2``. It will be first skipped directly by ``LatestOnlyOperator``, -even its ``trigger_rule`` is set to ``all_done``. +``task2``, but it will not be skipped, since its ``trigger_rule`` is set to +``all_done``. Review comment: I think this should also be added to the Updating.md (see my comment). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
feluelle commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#discussion_r369467873 ## File path: docs/concepts.rst ## @@ -1078,15 +1078,14 @@ For example, consider the following DAG: trigger_rule=TriggerRule.ALL_DONE) task4.set_upstream([task1, task2]) -In the case of this DAG, the ``latest_only`` task will show up as skipped -for all runs except the latest run. ``task1`` is directly downstream of -``latest_only`` and will also skip for all runs except the latest. +In the case of this DAG, the task ``task1`` is directly downstream of +``latest_only`` and will be skipped for all runs except the latest. ``task2`` is entirely independent of ``latest_only`` and will run in all scheduled periods. ``task3`` is downstream of ``task1`` and ``task2`` and because of the default ``trigger_rule`` being ``all_success`` will receive a cascaded skip from ``task1``. ``task4`` is downstream of ``task1`` and -``task2``. It will be first skipped directly by ``LatestOnlyOperator``, -even its ``trigger_rule`` is set to ``all_done``. +``task2``, but it will not be skipped, since its ``trigger_rule`` is set to +``all_done``. Review comment: That is something users should be aware of when updating Airflow. Don't you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable
kaxil commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable URL: https://github.com/apache/airflow/pull/7143#discussion_r369470733 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -537,6 +537,9 @@ ssl_cacert = # https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html pool = prefork +# The number of seconds to wait before timing out apply_async +connection_timeout = 2 Review comment: Is it actually a `connection_timeout`? Trying to think if we can call it something specific to apply_async This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable
kaxil commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable URL: https://github.com/apache/airflow/pull/7143#discussion_r369470809 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -537,6 +537,9 @@ ssl_cacert = # https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html pool = prefork +# The number of seconds to wait before timing out apply_async +connection_timeout = 2 Review comment: @ashb what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
kaxil commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#discussion_r369471993 ## File path: docs/concepts.rst ## @@ -1078,15 +1078,14 @@ For example, consider the following DAG: trigger_rule=TriggerRule.ALL_DONE) task4.set_upstream([task1, task2]) -In the case of this DAG, the ``latest_only`` task will show up as skipped -for all runs except the latest run. ``task1`` is directly downstream of -``latest_only`` and will also skip for all runs except the latest. +In the case of this DAG, the task ``task1`` is directly downstream of +``latest_only`` and will be skipped for all runs except the latest. ``task2`` is entirely independent of ``latest_only`` and will run in all scheduled periods. ``task3`` is downstream of ``task1`` and ``task2`` and because of the default ``trigger_rule`` being ``all_success`` will receive a cascaded skip from ``task1``. ``task4`` is downstream of ``task1`` and -``task2``. It will be first skipped directly by ``LatestOnlyOperator``, -even its ``trigger_rule`` is set to ``all_done``. +``task2``, but it will not be skipped, since its ``trigger_rule`` is set to +``all_done``. Review comment: Agree with @feluelle , let's add this to Updating.md Good work @m1racoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable
yuqian90 commented on a change in pull request #7143: [AIRFLOW-6527] Make send_task_to_executor timeout configurable URL: https://github.com/apache/airflow/pull/7143#discussion_r369478045 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -537,6 +537,9 @@ ssl_cacert = # https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html pool = prefork +# The number of seconds to wait before timing out apply_async +connection_timeout = 2 Review comment: You are right. To be precise, this will become the timeout for `fetch_celery_task_state()` and `send_task_to_executor()`. I could not come up with a perfect name for it. I'm open to suggestions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream
kaxil commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream URL: https://github.com/apache/airflow/pull/5308#issuecomment-577115221 I have resolved the conflicts @potiuk and @TeddyHartanto . I think we can merge it as it is. Already has an approval, any objections, comment @potiuk or @ashb ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream
kaxil edited a comment on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream URL: https://github.com/apache/airflow/pull/5308#issuecomment-577115221 I have resolved the conflicts @potiuk and @TeddyHartanto . I think we can merge it as it is. Already has an approval, any objections, comment @potiuk or @ashb ? Although, let's wait for the CI results. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream
kaxil edited a comment on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream URL: https://github.com/apache/airflow/pull/5308#issuecomment-577115221 I have resolved the conflicts @potiuk and @TeddyHartanto . I think we can merge it as it is. Already has an approval, any objections, comment @potiuk or @ashb ? Although, let's wait for the CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #2460: [AIRFLOW-1424] make the next execution date of DAGs visible
kaxil commented on a change in pull request #2460: [AIRFLOW-1424] make the next execution date of DAGs visible URL: https://github.com/apache/airflow/pull/2460#discussion_r369484249 ## File path: airflow/jobs.py ## @@ -892,6 +891,11 @@ def create_dag_run(self, dag, session=None): if next_run_date and min_task_end_date and next_run_date > min_task_end_date: return +# Don't really schedule the job, we are interested in its next run date +# as calculated by the scheduler +if dry_run is True: +return next_run_date Review comment: ping @ashb and @ultrabug . Can I help you guys to get this over the line? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook
kaxil commented on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook URL: https://github.com/apache/airflow/pull/6457#issuecomment-577120725 > https://github.com/apache/airflow/blob/master/airflow/providers/apache/pinot/hooks/pinot.py#L217 It is actually used. It is the way the **DbApiHook** is implemented. https://github.com/apache/airflow/blob/1fa7ac1b3978469ba3ec9b902b4f2b4890f8199d/airflow/hooks/dbapi_hook.py#L36-L57 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook
kaxil commented on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook URL: https://github.com/apache/airflow/pull/6457#issuecomment-577125000 I have resolved the merge conflict, waiting for the CI before I merge. Although @sekikn can you add a test to check if the connection is created? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6616) Add Tableau Hook and Operators
Rok Mihevc created AIRFLOW-6616: --- Summary: Add Tableau Hook and Operators Key: AIRFLOW-6616 URL: https://issues.apache.org/jira/browse/AIRFLOW-6616 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Affects Versions: 2.0.0 Reporter: Rok Mihevc Fix For: 2.0.0 Tableau (tableau.com) provides a python client (github.com/tableau/server-client-python) that enables automation of certain tasks (e.g. report sending). Addition of Tableau hook and operators will enable using Tableau to generate and send as steps in a wider ETL process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil commented on a change in pull request #6900: [AIRFLOW-6346] Enhance dag default_view and orientation
kaxil commented on a change in pull request #6900: [AIRFLOW-6346] Enhance dag default_view and orientation URL: https://github.com/apache/airflow/pull/6900#discussion_r369496408 ## File path: airflow/models/dag.py ## @@ -296,8 +298,16 @@ def __init__( self.max_active_runs = max_active_runs self.dagrun_timeout = dagrun_timeout self.sla_miss_callback = sla_miss_callback -self._default_view = default_view Review comment: I have no idea tbh, @vardancse @milton0825 - Do you guys have any idea about it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7230: [AIRFLOW-6602]: render executor_config before triggering tasks to su…
nuclearpinguin commented on issue #7230: [AIRFLOW-6602]: render executor_config before triggering tasks to su… URL: https://github.com/apache/airflow/pull/7230#issuecomment-577130919 Travis is sad, can you fix pylint errors? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7213: [AIRFLOW-6424] Added a operator to modify EMR cluster
nuclearpinguin commented on a change in pull request #7213: [AIRFLOW-6424]
Added a operator to modify EMR cluster
URL: https://github.com/apache/airflow/pull/7213#discussion_r369500126
##
File path: airflow/providers/amazon/aws/operators/emr_modify_cluster.py
##
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr import EmrHook
+from airflow.utils.decorators import apply_defaults
+
+
+class EmrModifyClusterOperator(BaseOperator):
+"""
+An operator that modifies an existing EMR cluster.
+:param cluster_id: cluster identifier
+:type cluster_id: str
+:param step_concurrency_level: Concurrency of the cluster
+:type step_concurrency_level: int
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+:param do_xcom_push: if True, cluster_id is pushed to XCom with key
cluster_id.
+:type do_xcom_push: bool
+"""
+template_fields = ['cluster_id', 'step_concurrency_level']
+template_ext = ()
+ui_color = '#f9c915'
+
+@apply_defaults
+def __init__(
+self,
+cluster_id: str,
+step_concurrency_level: int,
+aws_conn_id: str = 'aws_default',
+*args, **kwargs):
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use
'do_xcom_push' instead")
+if cluster_id is None or step_concurrency_level is None:
+raise AirflowException("'cluster_id' or 'step_concurrency_level'
not passed")
Review comment:
This will never happen as `cluster_id` and 'step_concurrency_level` are
required parameters. If user don't pass them it will raise TypeError, so we can
remove this statement.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
nuclearpinguin commented on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-577133637 > The test failures in Travis are in seemingly unrelated tests such as testviews.py. it's very unlikely to be caused by the new tests you added. How about trying to rerun or ask if anyone else see similar failure on slack? I restarted the tests to check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
kaxil commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139#issuecomment-577136743 The tests are unstable on v1-10-test branch. We are fixing it. Thanks for this backport PR @yuqian90 🎉 , really appreciate it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
kaxil edited a comment on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139#issuecomment-577136743 The tests are unstable on v1-10-test branch. We are fixing it, will keep you updated. Thanks for this backport PR @yuqian90 🎉 , really appreciate it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook
codecov-io edited a comment on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook URL: https://github.com/apache/airflow/pull/6457#issuecomment-552073991 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=h1) Report > Merging [#6457](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0cb9d2d124939ba19c539a8056ad5396c6f2dd39?src=pr&el=desc) will **increase** coverage by `1.29%`. > The diff coverage is `76.62%`. [](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6457 +/- ## == + Coverage 83.76% 85.05% +1.29% == Files 635 791 +156 Lines 3685140125+3274 == + Hits3086734128+3261 - Misses 5984 5997 +13 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/pubsub\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9wdWJzdWJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5) | `75% <ø> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (-2.97%)` | :arrow_down: | | [...w/example\_dags/example\_external\_task\_marker\_dag.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9leHRlcm5hbF90YXNrX21hcmtlcl9kYWcucHk=) | `100% <ø> (ø)` | | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `92.48% <ø> (+10.3%)` | :arrow_up: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/sensors/sagemaker\_tuning\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3R1bmluZ19zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...flow/contrib/sensors/sagemaker\_transform\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYW5zZm9ybV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | ... and [744 more](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=footer). Last update [0cb9d2d...8951bd1](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook
codecov-io edited a comment on issue #6457: [AIRFLOW-5800] Add a default connection entry for PinotDbApiHook URL: https://github.com/apache/airflow/pull/6457#issuecomment-552073991 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=h1) Report > Merging [#6457](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0cb9d2d124939ba19c539a8056ad5396c6f2dd39?src=pr&el=desc) will **increase** coverage by `1.29%`. > The diff coverage is `76.62%`. [](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6457 +/- ## == + Coverage 83.76% 85.05% +1.29% == Files 635 791 +156 Lines 3685140125+3274 == + Hits3086734128+3261 - Misses 5984 5997 +13 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/pubsub\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9wdWJzdWJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5) | `75% <ø> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (-2.97%)` | :arrow_down: | | [...w/example\_dags/example\_external\_task\_marker\_dag.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9leHRlcm5hbF90YXNrX21hcmtlcl9kYWcucHk=) | `100% <ø> (ø)` | | | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `92.48% <ø> (+10.3%)` | :arrow_up: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/contrib/sensors/sagemaker\_tuning\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3R1bmluZ19zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...flow/contrib/sensors/sagemaker\_transform\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYW5zZm9ybV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | ... and [744 more](https://codecov.io/gh/apache/airflow/pull/6457/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=footer). Last update [0cb9d2d...8951bd1](https://codecov.io/gh/apache/airflow/pull/6457?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@b5bd9ab`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `97.77%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#7038 +/- ## = Coverage ? 83.36% = Files ? 791 Lines ?41468 Branches ?0 = Hits ?34568 Misses? 6900 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.9% <ø> (ø)` | | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.52% <ø> (ø)` | | | [...ib/operators/azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9jb250YWluZXJfaW5zdGFuY2VzX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.28% <ø> (ø)` | | | [airflow/ti\_deps/deps/prev\_dagrun\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcHJldl9kYWdydW5fZGVwLnB5) | `80.55% <ø> (ø)` | | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <0%> (ø)` | | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.5% <0%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <100%> (ø)` | | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=footer). Last update [b5bd9ab...40d17ba](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@b5bd9ab`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `97.77%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#7038 +/- ## = Coverage ? 83.36% = Files ? 791 Lines ?41468 Branches ?0 = Hits ?34568 Misses? 6900 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.9% <ø> (ø)` | | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.52% <ø> (ø)` | | | [...ib/operators/azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9jb250YWluZXJfaW5zdGFuY2VzX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.28% <ø> (ø)` | | | [airflow/ti\_deps/deps/prev\_dagrun\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcHJldl9kYWdydW5fZGVwLnB5) | `80.55% <ø> (ø)` | | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <0%> (ø)` | | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.5% <0%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <100%> (ø)` | | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=footer). Last update [b5bd9ab...40d17ba](https://codecov.io/gh/apache/airflow/pull/7038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#issuecomment-577144219 @nuclearpinguin made the changes you requested: [Changes after CR 3](https://github.com/apache/airflow/pull/6824/commits/3dce509a1c58c4f9cb0dd62a413e4f14ee40c987) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] roitvt commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor
roitvt commented on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor URL: https://github.com/apache/airflow/pull/7163#issuecomment-577146596 ping @mik-laj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6583) (BigQuery) Add query_params to templated_fields
[
https://issues.apache.org/jira/browse/AIRFLOW-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jithin Sukumar updated AIRFLOW-6583:
Description:
To query time-partitioned tables, I am passing {{query_params}} like this
yesterday = Variable.get('yesterday', 'yesterday_ds')
today = Variable.get('today', 'ds')
...
query_params=[{'name': 'yesterday', 'parameterType':
{'type': 'STRING'}
,
'parameterValue': \{'value': yesterday}},
{'name': 'today', 'parameterType':
{'type': 'STRING'}
,
'parameterValue': \{'value': today}}]
query_params needs to be a template_field
was:
To query time-partitioned tables, I am passing \{{query_params}} like this
yesterday = Variable.get('yesterday', '\{{yesterday_ds}}')
today = Variable.get('today', '\{{ds}}')
...
query_params=[\{'name': 'yesterday', 'parameterType': {'type': 'STRING'},
'parameterValue': \{'value': yesterday}},
\{'name': 'today', 'parameterType': {'type': 'STRING'},
'parameterValue': \{'value': today}}]
query_params needs to be a template_field
> (BigQuery) Add query_params to templated_fields
> ---
>
> Key: AIRFLOW-6583
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6583
> Project: Apache Airflow
> Issue Type: New Feature
> Components: gcp
>Affects Versions: 1.10.7
>Reporter: Jithin Sukumar
>Assignee: Jithin Sukumar
>Priority: Minor
> Fix For: 2.0.0
>
>
> To query time-partitioned tables, I am passing {{query_params}} like this
> yesterday = Variable.get('yesterday', 'yesterday_ds')
> today = Variable.get('today', 'ds')
> ...
> query_params=[{'name': 'yesterday', 'parameterType':
> {'type': 'STRING'}
> ,
> 'parameterValue': \{'value': yesterday}},
> {'name': 'today', 'parameterType':
> {'type': 'STRING'}
> ,
> 'parameterValue': \{'value': today}}]
> query_params needs to be a template_field
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7227: [AIRFLOW-6539] Allow Custom Statsd Client
codecov-io commented on issue #7227: [AIRFLOW-6539] Allow Custom Statsd Client URL: https://github.com/apache/airflow/pull/7227#issuecomment-577151472 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=h1) Report > Merging [#7227](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5b44c0fc4ac0030c78ab6ad4e658e9abdf60dc04?src=pr&el=desc) will **decrease** coverage by `0.49%`. > The diff coverage is `9.09%`. [](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#7227 +/- ## = - Coverage 85.53% 85.04% -0.5% = Files 756 791 +35 Lines 3985940134+275 = + Hits3409434132 +38 - Misses 5765 6002+237 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/stats.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9zdGF0cy5weQ==) | `64.44% <9.09%> (-6.81%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: | | [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [airflow/api/client/local\_client.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L2xvY2FsX2NsaWVudC5weQ==) | `92% <0%> (-8%)` | :arrow_down: | | [airflow/api/client/api\_client.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L2FwaV9jbGllbnQucHk=) | `63.15% <0%> (-1.55%)` | :arrow_down: | | ... and [78 more](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=footer). Last update [5b44c0f...fb21b77](https://codecov.io/gh/apache/airflow/pull/7227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
tooptoop4 commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-577157255 @kaxil pls merge :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 removed a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
tooptoop4 removed a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-572718287 @ashb pls merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
kaxil merged pull request #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'
[
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-4495.
-
Fix Version/s: 1.10.8
2.0.0
Resolution: Fixed
> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> 1.
> useful to handle future date for externally triggered batch process where
> ingesting 'forecast' data where filename date is in the future
> 2.
> this error is just in the scheduler log and not propagated up, so the dag
> stays in 'running' state forever (or for 1 year waiting for the time to pass
> :) )
> ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>
>
> fix below works if u only have externally triggered DAGs:
>
> commenting below ti_deps\deps\runnable_exec_date_dep.py
> #if ti.execution_date > cur_date:
> # yield self._failing_status(
> # reason="Execution date \{0} is in the future (the current "
> # "date is \{1}).".format(ti.execution_date.isoformat(),
> # cur_date.isoformat()))
>
> commenting below jobs.py
> # don't consider runs that are executed in the future
> #if run.execution_date > timezone.utcnow():
> # self.log.error(
> # "Execution date is in future: %s",
> # run.execution_date
> # )
> # continue
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'
[
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021007#comment-17021007
]
ASF GitHub Bot commented on AIRFLOW-4495:
-
kaxil commented on pull request #7038: [AIRFLOW-4495] Allow externally
triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
>
> 1.
> useful to handle future date for externally triggered batch process where
> ingesting 'forecast' data where filename date is in the future
> 2.
> this error is just in the scheduler log and not propagated up, so the dag
> stays in 'running' state forever (or for 1 year waiting for the time to pass
> :) )
> ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>
>
> fix below works if u only have externally triggered DAGs:
>
> commenting below ti_deps\deps\runnable_exec_date_dep.py
> #if ti.execution_date > cur_date:
> # yield self._failing_status(
> # reason="Execution date \{0} is in the future (the current "
> # "date is \{1}).".format(ti.execution_date.isoformat(),
> # cur_date.isoformat()))
>
> commenting below jobs.py
> # don't consider runs that are executed in the future
> #if run.execution_date > timezone.utcnow():
> # self.log.error(
> # "Execution date is in future: %s",
> # run.execution_date
> # )
> # continue
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4495) allow externally triggered dags to run for future 'Execution date'
[
https://issues.apache.org/jira/browse/AIRFLOW-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021008#comment-17021008
]
ASF subversion and git services commented on AIRFLOW-4495:
--
Commit 6414d38a04d74b04bd3c0add1dd61fff3df740c1 in airflow's branch
refs/heads/master from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6414d38 ]
[AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
(#7038)
> allow externally triggered dags to run for future 'Execution date'
> --
>
> Key: AIRFLOW-4495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4495
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
> Fix For: 2.0.0, 1.10.8
>
>
> 1.
> useful to handle future date for externally triggered batch process where
> ingesting 'forecast' data where filename date is in the future
> 2.
> this error is just in the scheduler log and not propagated up, so the dag
> stays in 'running' state forever (or for 1 year waiting for the time to pass
> :) )
> ERROR - Execution date is in future: 2020-01-01 00:00:00+00:00
>
>
> fix below works if u only have externally triggered DAGs:
>
> commenting below ti_deps\deps\runnable_exec_date_dep.py
> #if ti.execution_date > cur_date:
> # yield self._failing_status(
> # reason="Execution date \{0} is in the future (the current "
> # "date is \{1}).".format(ti.execution_date.isoformat(),
> # cur_date.isoformat()))
>
> commenting below jobs.py
> # don't consider runs that are executed in the future
> #if run.execution_date > timezone.utcnow():
> # self.log.error(
> # "Execution date is in future: %s",
> # run.execution_date
> # )
> # continue
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6617) Add tests to PYTHONPATH in breeze
Tomasz Urbaszek created AIRFLOW-6617: Summary: Add tests to PYTHONPATH in breeze Key: AIRFLOW-6617 URL: https://issues.apache.org/jira/browse/AIRFLOW-6617 Project: Apache Airflow Issue Type: Improvement Components: breeze, tests Affects Versions: 1.10.8 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin opened a new pull request #7241: [AIRFLOW-6617] Add tests to PYTHONPATH in breeze
nuclearpinguin opened a new pull request #7241: [AIRFLOW-6617] Add tests to PYTHONPATH in breeze URL: https://github.com/apache/airflow/pull/7241 Adding tests directory to PYTHONPATH helps during development :) --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6617) Add tests to PYTHONPATH in breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021035#comment-17021035 ] ASF GitHub Bot commented on AIRFLOW-6617: - nuclearpinguin commented on pull request #7241: [AIRFLOW-6617] Add tests to PYTHONPATH in breeze URL: https://github.com/apache/airflow/pull/7241 Adding tests directory to PYTHONPATH helps during development :) --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add tests to PYTHONPATH in breeze > - > > Key: AIRFLOW-6617 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6617 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6551) KubernetesExecutor does not create dynamic pods for tasks inside subdag
[
https://issues.apache.org/jira/browse/AIRFLOW-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hamid Mahmood reassigned AIRFLOW-6551:
--
Assignee: (was: Daniel Imberman)
> KubernetesExecutor does not create dynamic pods for tasks inside subdag
> ---
>
> Key: AIRFLOW-6551
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6551
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor-kubernetes
>Affects Versions: 1.10.7
>Reporter: Hamid Mahmood
>Priority: Major
>
> KubernetesExecutor does not create dynamic pods for tasks inside subdag.
> I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators
> inside a main_dag. Following is the hierarchy
> * main_dag:
> ** subdagA
> *** taskA1
> *** taskA2
> *** taskA3
> ** subdagB
> *** subdagB_1
> taskB1
> taskB2
> ** task_main1
> ** task_main2
> ** task_main3
> I have tested the following three scenarios.
> *Scenario:1*
> I have set the following parameter
> {code:java}
> AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
> {code}
> When I run the workflow main_dag, only a single pod is created with name
> subdagA-
> 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3)
> run inside that single pod.
> In theory there should be 3 separate pods for these 3 tasks inside subdagA.
> Same is the case for subdagB, only a single pod is created to run subdagB,
> subdagB_1 and its tasks runs inside that pod.
> But the tasks (task_main1, task_main2, task_main3) that are not inside
> further subdag runs in dynamic pods.
>
> *Scenario: 2*
> I have set the following
> {code:java}
> AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
> {code}
> and passed executor=KubernetesExecutor() when creating the subdag.
> {code:java}
> SubDagOperator( task_id=task_id, subdag=subdag, dag=parent_dag,
> executor=KubernetesExecutor(), **kwargs )
> {code}
> Still one pod is created for SubtaskA but now the taskA1 inside subdagA got
> stuck in queue state, when I checked the logs of this pod I got the following
> {code:java}
> Running %s on host %s 2020-01-12T02:15:00+00:00 [queued]>
> maindagsubdagA-1a31ef3e2aef489daf1b329160646{code}
> *Scenario: 3*
> I have set the following
> {code:java}
> AIRFLOW__CORE__EXECUTOR = LocalExecutor
> {code}
> and passed executor=KubernetesExecutor() when creating the subdag.
> {code:java}
> SubDagOperator( task_id=task_id, subdag=subdag, dag=parent_dag,
> executor=KubernetesExecutor(), **kwargs )
> {code}
> Now this time dynamic pods are created for single task inside subdagA and
> subdagB and I get the parallelism. In this case KubernetesExecutor shows the
> required behavior. But the downside of this is that these three tasks
> task_main1, task_main2 and task_main3 will use LocalExecutor and will run in
> scheduler pod.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369587280
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id).get_conn())
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
Review comment:
Agree with @nuclearpinguin
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369586964
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#6b9659'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(params=params, aws_conn_id=aws_conn_id, *args,
**kwargs)
+self.stack_name = stack_name
+
+def cloudformation_op(self, cloudformation):
+cloudformation.create_stack(self.stack_name, self.params)
+
+
+class CloudFormationDeleteStackOperator(BaseCloudFormationOperator):
+"""
+An operator that deletes a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.delete_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params=None,
+aws_conn_id='aws_default',
+*args, **kwargs):
+if params is None:
+params = {}
Review comment:
`self.params = params or {}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369590133
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#6b9659'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(params=params, aws_conn_id=aws_conn_id, *args,
**kwargs)
+self.stack_name = stack_name
+
+def cloudformation_op(self, cloudformation):
+cloudformation.create_stack(self.stack_name, self.params)
+
+
+class CloudFormationDeleteStackOperator(BaseCloudFormationOperator):
+"""
+An operator that deletes a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.delete_stack
Review comment:
Same here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369589043
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class CloudFormationCreateStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be created successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status == 'CREATE_COMPLETE':
+return True
+elif stack_status in ('CREATE_IN_PROGRESS', None):
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
+
+
+class CloudFormationDeleteStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be deleted successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status in ('DELETE_COMPLETE', None):
+return True
+elif stack_status == 'DELETE_IN_PROGRESS':
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
Review comment:
Same could be done here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369593236
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
```python
except ClientError as e:
if 'does not exist' in str(e):
return None
raise e
```
I am also wondering if `ClientError` is the correct exception type to catch.
The docs
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.describe_stacks
are saying that it raises an `AmazonCloudFormationException` when it does not
exist.
So then I would only catch this:
```python
except AmazonCloudFormationException as e:
return None
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369590507
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
Review comment:
You can use `.. seealso::` here.
https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html#colored-boxes-note-seealso-todo-and-warnings
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
feluelle commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369587843
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class CloudFormationCreateStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be created successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status == 'CREATE_COMPLETE':
+return True
+elif stack_status in ('CREATE_IN_PROGRESS', None):
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
Review comment:
```python
if stack_status == 'CREATE_COMPLETE':
return True
if stack_status in ('CREATE_IN_PROGRESS', None):
return False
raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6618) kubernetes_pod_operator no longer using kube context in provided config_file
Pat Carroll created AIRFLOW-6618:
Summary: kubernetes_pod_operator no longer using kube context in
provided config_file
Key: AIRFLOW-6618
URL: https://issues.apache.org/jira/browse/AIRFLOW-6618
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.7
Environment: Airflow on Kubernetes in named namespace (not default
namespace) and using a specified token - not default namespace token.
Reporter: Pat Carroll
Call KubernetesPodOperator and provide kubernetes context via a config_file.
KubernetesPodOperator uses default token for the namespace which, in our env,
does not have necessary privileges to create a pod. It fails to use token
specified in config_file.
Same occurs when using default kubernetes config in ${AIRFLOW_HOME}/.kube/config
This is a new issue that does not exist in 1.10.6.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369636858 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting Review comment: ```suggestion time we should make sure that the code can be taken over by others - without excessive communication. Setting ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369505708 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting +high requirements for the code (fairly strict code review, static code checks, requirements of automated +tests, pre-commit checks ) is the best way that we can achieve that - by only accepting good quality +code with full test coverage we make sure that we can work with the code in the future. +So do not be surprised if you are asked to add more tests or make the code cleaner - +this is for the sake of maintainability. + +Here are a few rules that are important to keep in mind when you enter our community: + + * Do not be afraid to ask questions + * The communication is asynchronous - do not expect immediate answers, ping others if you are blocked + * There is a #newbie-questions channel in slack as a safe place to ask questions + * You can ask one of the committers to be a mentor for you, committers can guide within the community Review comment: ```suggestion * You can ask one of the committers to be a mentor for you, committers can guide you through the community ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369638493 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting +high requirements for the code (fairly strict code review, static code checks, requirements of automated +tests, pre-commit checks ) is the best way that we can achieve that - by only accepting good quality Review comment: ```suggestion tests, pre-commit checks) is the best way to achieve that - by only accepting top-notch quality ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369121248 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. Review comment: ```suggestion This means that communication is very important, and this chapter is all about it. ``` This means that communication plays the big role in it, and this chapter is all about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369505260 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting +high requirements for the code (fairly strict code review, static code checks, requirements of automated +tests, pre-commit checks ) is the best way that we can achieve that - by only accepting good quality +code with full test coverage we make sure that we can work with the code in the future. +So do not be surprised if you are asked to add more tests or make the code cleaner - +this is for the sake of maintainability. + +Here are a few rules that are important to keep in mind when you enter our community: + + * Do not be afraid to ask questions + * The communication is asynchronous - do not expect immediate answers, ping others if you are blocked Review comment: ```suggestion * The communication is asynchronous - do not expect immediate answers, ping others if you are blocked ``` Where can I ping someone? Devlist / Slack? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369634841 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those Review comment: ```suggestion a consensus, conclusions or we end up voting to make certain decisions. It is important that these ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369121587 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments Review comment: ```suggestion We have various channel of communication - starting from the official devlist, comments ``` We have various channels of communication - starting from the official devlist, comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369503677 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,63 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than the code they create. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations Review comment: I know this conversation is marked as resolved but I agree with @nuclearpinguin here. At the moment, a message sounds as if committers or PMC members were gods and we shouldn't bother them because they have private lives 😄 Everyone does have private life / duties etc. In general I'd make the whole paragraph shorter and more concise. The core message in my opinion should be about: ```suggestion We are all devoting our time for community as individuals who except for being active in Apache Airflow have families, daily jobs, right for vacation. Sometimes we are in different time zones or simply are busiy with day-to-day duties that our response time might be delayed. For us it's crucial to remember to respect each other in the project with no formal structure. There are no managers, departments, most of us is autonomous in our opinions, decisions. All of it makes Apache Airflow community a great space for open discussion and mutual respect for various opinions. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369505507 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting +high requirements for the code (fairly strict code review, static code checks, requirements of automated +tests, pre-commit checks ) is the best way that we can achieve that - by only accepting good quality +code with full test coverage we make sure that we can work with the code in the future. +So do not be surprised if you are asked to add more tests or make the code cleaner - +this is for the sake of maintainability. + +Here are a few rules that are important to keep in mind when you enter our community: + + * Do not be afraid to ask questions + * The communication is asynchronous - do not expect immediate answers, ping others if you are blocked + * There is a #newbie-questions channel in slack as a safe place to ask questions Review comment: Ok, thanks 😄I think I'd mention this above when you write about Slack as a channel of communcation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369639040 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any +time we should be sure that the code can be taken over by others - without excessive communication. Setting +high requirements for the code (fairly strict code review, static code checks, requirements of automated +tests, pre-commit checks ) is the best way that we can achieve that - by only accepting good quality +code with full test coverage we make sure that we can work with the code in the future. Review comment: ```suggestion code. Thanks to full test coverage we can make sure that we will be able to work with the code in the future. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369636196 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add Review comment: ```suggestion and loosely organised team - we have both contributors that commit one commit only and people who add ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369635061 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice Review comment: ```suggestion and what's good for community, should be accepted even if you have a different opinion. There is a nice ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369636735 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. + +Disagreements are expected, discussions might include strong opinions and contradicting statements. +Sometimes you might get two committers asking you to do things differently. This all happened in the past +and will continue to happen. As a community we have some mechanisms to facilitate discussion and come to +a consensus, conclusions and sometime we end up voting certain decisions. It is important that those +decisions are not treated as personal wins or looses. At the end it's the community that we all care about +and what's good for community, should be accepted even if you have different opinion. There is the nice +motto that you should follow in case you disagree with community decision "Disagree but engage". Even +if you do not agree with a community decision, you should follow it and embrace (but you are free to +express your opinion that you don't agree with it). + +As a community - we have high requirements for code quality. This is mainly because we are a distributed +and loosely organised team - sometime we have contributors that commit one commit only, sometimes people add +more commits, sometimes people assume informal "stewardship" over parts of code for some time - but at any Review comment: ```suggestion more commits. It happens that some people assume informal "stewardship" over parts of code - but at any ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369500411 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. Review comment: [UI] I'm not sure how is it going to look in terms of the final UI but I'd use bullet points for each channel. Let's say it's "Wiki", so I'd describe the purpose of Wiki below in bullet points: - detailed discussions on big proposals (AIPs) - helpful shared resources (what type of resources btw?) Maybe it's not that useful in terms of description of the subject for Wiki but for Slack for sure. Let me know what you think. [Content] **Devlist** If I were a beginner contributor I wouldn't know what actually dev-list is. What is lacking personally to me are the rules for communicating within the devlist itself. By this I mean that I don't know how I should compose the subject when I write about something and if I should add [DISCUSSION] / [AIP-XY] etc. It's a general comment that I'd also like to add to the Apache Airflow devlist. The reason I am not active within the devlist is my lack of knowledge on how to use it, so that everyone understands my point of view, the subject I touch is well-classified etc. **Slack** How do I get an invitation to Slack? What is Slack? Some people might don't know. For most of you it sounds obvious but I'd be kind of lost. Speaking of China e.g. they have so different communication tools that I believe it might be worth to shortly describe Slack as a chat tool for ... _If you want to join Slack, ask XYZ for an invitation blah blah blah_ Hope the above message is clear, what's your opinion? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369503819 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals +that contribute a lot of their free time on top of their work time to develop Apache Airflow. +They have families, personal life, sometimes they are swamped with work, sometimes they are on vacations +sometimes they are simply tired and need a time off. They are in various timezones and their responsiveness +will vary from time to time. As a community project we have no formal structure, we have no managers nor +departments and most of the people are autonomous in their opinions and decisions, so sometimes even +committers will have different opinions on the same subjects. Review comment: See my comment from the above discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mschickensoup commented on a change in pull request #7204: [AIRFLOW-XXXX] - add communication chapter to contributing
mschickensoup commented on a change in pull request #7204: [AIRFLOW-] - add communication chapter to contributing URL: https://github.com/apache/airflow/pull/7204#discussion_r369122288 ## File path: CONTRIBUTING.rst ## @@ -664,6 +664,73 @@ Later on above depending if you have more commits that cause conflicts in your PR (rebasing applies each commit from your PR one-by-one). +How to communicate +== + +Apache Airflow is a Community within Apache Software Foundation. As the motto of +the Apache Software Foundation states "Community over Code" - people in the +community are far more important than their contribution. + +This means that communication is very important, and this chapter is all about it. + +We have various channel of communication - starting from the official devlist, comments +in the Pull Requests, Slack, wiki. + +All those channels can be used for different purposes: + +* The devlist for official communication, discussing proposals, voting, general issues, asking community for + opinion etc. +* Wiki for detailed discussions on big proposals (AIPs) and some helpful, shared resources. +* PRs for discussing implementation details of this PR (not for architectural discussions) +* Slack for ad-hoc questions, troubleshooting, occassional discussions, group talks (including SIG - special +interest groups), notifications, random queries, newbie questions etc. + + +It is important to remember that all contributors, committers and even PMC members are individuals Review comment: even PMC members? I'd delete "even" :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] notatallshaw commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates
notatallshaw commented on issue #7038: [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-577275156 Sorry for the comment spam but I thought it would be important for @yuqian90 We have the exact same problem you have, we solve it by setting the DAG with a start date of the timezones it should be scheduled in, e.g. Tokyo time. We then create a macro which is a modified version of this: https://stackoverflow.com/a/54922776/2958068 The macro above provides the execution date in the same time zone as the DAG. Further we modify it to provide the actual scheduled date e.g. if scheduled to run on 2019-01-01 08:00 that's the datetime it provides (which in the default Airlow variables is the 'next execution date'). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman merged pull request #7236: [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg
dimberman merged pull request #7236: [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg URL: https://github.com/apache/airflow/pull/7236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6611) Add missing ProxyFix configs to default_airflow.cfg with docs
[ https://issues.apache.org/jira/browse/AIRFLOW-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021277#comment-17021277 ] ASF subversion and git services commented on AIRFLOW-6611: -- Commit c890d066965aa9dbf3016f41cfae45e9a084478a in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c890d06 ] [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236) > Add missing ProxyFix configs to default_airflow.cfg with docs > - > > Key: AIRFLOW-6611 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6611 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, documentation >Affects Versions: 1.10.7 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.8 > > > https://github.com/apache/airflow/commit/d90ddbd189e4ef99d45af44abf8580b29896a4e0 > added some new configs which weren't documented and weren't added in > default_airflow.cfg -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6611) Add missing ProxyFix configs to default_airflow.cfg with docs
[ https://issues.apache.org/jira/browse/AIRFLOW-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021275#comment-17021275 ] ASF GitHub Bot commented on AIRFLOW-6611: - dimberman commented on pull request #7236: [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg URL: https://github.com/apache/airflow/pull/7236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add missing ProxyFix configs to default_airflow.cfg with docs > - > > Key: AIRFLOW-6611 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6611 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, documentation >Affects Versions: 1.10.7 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.8 > > > https://github.com/apache/airflow/commit/d90ddbd189e4ef99d45af44abf8580b29896a4e0 > added some new configs which weren't documented and weren't added in > default_airflow.cfg -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6611) Add missing ProxyFix configs to default_airflow.cfg with docs
[ https://issues.apache.org/jira/browse/AIRFLOW-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-6611. - Resolution: Fixed > Add missing ProxyFix configs to default_airflow.cfg with docs > - > > Key: AIRFLOW-6611 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6611 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, documentation >Affects Versions: 1.10.7 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.8 > > > https://github.com/apache/airflow/commit/d90ddbd189e4ef99d45af44abf8580b29896a4e0 > added some new configs which weren't documented and weren't added in > default_airflow.cfg -- This message was sent by Atlassian Jira (v8.3.4#803005)
[airflow-on-k8s-operator] branch master updated: Add Repo name to Readme (#4)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow-on-k8s-operator.git The following commit(s) were added to refs/heads/master by this push: new 830f756 Add Repo name to Readme (#4) 830f756 is described below commit 830f7564c8b754539ecc29864ae350661a26f2d6 Author: Tao Feng AuthorDate: Wed Jan 22 09:25:31 2020 -0800 Add Repo name to Readme (#4) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f1cb1de..46c237a 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ +# Airflow On K8S Operator [](https://goreportcard.com/report/github.com/apache/airflow-on-k8s-operator) ## Community
[jira] [Commented] (AIRFLOW-6607) Get rid of old local_* scripts in Breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021335#comment-17021335 ] ASF subversion and git services commented on AIRFLOW-6607: -- Commit c1a09ec6f0ada279b52d792a1b8ba0b4be75fb03 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c1a09ec ] [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225) Breeze is now mature enought to remove old scripts that were doing the same what Breeze does now with command line. (cherry picked from commit 92d6e4a16f0c1a899d27f1e3e5d16873cf327152) > Get rid of old local_* scripts in Breeze > > > Key: AIRFLOW-6607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6607 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6592) Doc build can be run in parallel to tests in CI
[ https://issues.apache.org/jira/browse/AIRFLOW-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021333#comment-17021333 ] ASF subversion and git services commented on AIRFLOW-6592: -- Commit 6a63cf57007102cd58fb9a3a9362cd10d4d3eda3 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6a63cf5 ] [AIRFLOW-6592] Doc build is moved to test phase (#7208) (cherry picked from commit 37f026f66d0214449cad824d3d343999186d013e) > Doc build can be run in parallel to tests in CI > --- > > Key: AIRFLOW-6592 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6592 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6589) BATS tests are not executed when bash scripts change in pre-commit
[ https://issues.apache.org/jira/browse/AIRFLOW-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021334#comment-17021334 ] ASF subversion and git services commented on AIRFLOW-6589: -- Commit f0ec768b1001f65fd560cea7dd7256a236aa0fb1 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f0ec768 ] [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203) (cherry picked from commit e2524fac59df1a405a38f7f71b99436b44a47378) > BATS tests are not executed when bash scripts change in pre-commit > -- > > Key: AIRFLOW-6589 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6589 > Project: Apache Airflow > Issue Type: Improvement > Components: pre-commit >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6609) Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2
[
https://issues.apache.org/jira/browse/AIRFLOW-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021345#comment-17021345
]
Chris Schmautz commented on AIRFLOW-6609:
-
Either checking ahead of table existence before the addition, or dropping the
table ahead of the addition, works to mitigate the error.
Not sure why the error came up - resetdb numerous times at the lower revision
with little effect. I was able to step incrementally through revisions until
the 1.10.6 to 1.10.7 migration specifically.
> Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2
>
>
> Key: AIRFLOW-6609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6609
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.7
>Reporter: Chris Schmautz
>Priority: Major
> Labels: database, postgres
>
> We're attempting an upgrade from 1.10.3 to 1.10.7 to use some of the great
> features available in later revisions; however, the upgrade from 1.10.6 to
> 1.10.7 is causing some heartburn.
> +Runtime environment:+
> - Docker containers for each runtime segment (webserver, scheduler, flower,
> postgres, redis, worker)
> - Using CeleryExecutor queued with Redis
> - Using Postgres backend
>
> +Steps to reproduce:+
> 1. Author base images relating to each version of Airflow between 1.10.3 and
> 1.10.7 (if you want the full regression we have done)
> 2. 'airflow initdb' on revision 1.10.3
> 3. Start up the containers, run some dags, produce metadata
> 4. Increment / swap out base image revision from 1.10.3 base to 1.10.4 base
> image
> 5. Run 'airflow upgradedb'
> 6. Validate success
> n. Eventually you will get to the 1.10.6 revision, stepping up to 1.10.7,
> which produces the error below
>
> {code:java}
> INFO [alembic.runtime.migration] Running upgrade 6e96a59344a4 ->
> d38e04c12aa2, add serialized_dag table
> Revision ID: d38e04c12aa2
> Revises: 6e96a59344a4
> Create Date: 2019-08-01 14:39:35.616417
> Traceback (most recent call last):
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
> line 1246, in _execute_context
> cursor, statement, parameters, context
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
> line 581, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.errors.DuplicateTable: relation "serialized_dag" already exists
> The above exception was the direct cause of the following exception:Traceback
> (most recent call last):
> File "/opt/anaconda/miniconda3/envs/airflow/bin/airflow", line 37, in
>
> args.func(args)
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py",
> line 75, in wrapper
> return f(*args, **kwargs)
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py",
> line 1193, in upgradedb
> db.upgradedb()
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py",
> line 376, in upgradedb
> command.upgrade(config, 'heads')
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/command.py",
> line 298, in upgrade
> script.run_env()
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py",
> line 489, in run_env
> util.load_python_file(self.dir, "env.py")
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py",
> line 98, in load_python_file
> module = load_module_py(module_id, path)
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py",
> line 173, in load_module_py
> spec.loader.exec_module(module)
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 96, in
> run_migrations_online()
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 90, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/environment.py",
> line 846, in run_migrations
> self.get_context().run_migrations(**kw)
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 518, in run_migrations
> step.migration_fn(**kw)
> File
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migra
[GitHub] [airflow] potiuk commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream
potiuk commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream URL: https://github.com/apache/airflow/pull/5308#issuecomment-577317726 No objections. Just static checks failure :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7241: [AIRFLOW-6617] Add tests to PYTHONPATH in breeze
potiuk commented on issue #7241: [AIRFLOW-6617] Add tests to PYTHONPATH in breeze URL: https://github.com/apache/airflow/pull/7241#issuecomment-577319557 Just wondering how it helps ? Not that I am against it but like to see a reason for it and maybe add it for others to the docs how it can be helpful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6496) Choosing integration at Breeze start
[ https://issues.apache.org/jira/browse/AIRFLOW-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021375#comment-17021375 ] ASF subversion and git services commented on AIRFLOW-6496: -- Commit c5601aeddc37cbbc8911a5a68490752d538ea0a5 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c5601ae ] fixup! [AIRFLOW-6496] Separate integrations in tests (#7091) > Choosing integration at Breeze start > > > Key: AIRFLOW-6496 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6496 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > > You should be able to choose which integration you want to start when you run > Breeze. > At later stage we will ad markers to tests to run them only when given > environment is available.. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4113) Unpin boto library
[ https://issues.apache.org/jira/browse/AIRFLOW-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021398#comment-17021398 ] ASF subversion and git services commented on AIRFLOW-4113: -- Commit adcebe2b6813f89c9c1c708fb9f55b1cc67626ac in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=adcebe2 ] fixup! [AIRFLOW-4113] Unpin boto3 (#6884) > Unpin boto library > -- > > Key: AIRFLOW-4113 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4113 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.2 >Reporter: Fokko Driesprong >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0, 1.10.8 > > > Right now we're pinning boto on <1.8 because of an underlying issue in moto. > This issue has been solved, and therefore it would be nice to update again to > a more recent version of boto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-577337258 [Here are my thoughts](https://gist.github.com/vsoch/599bd51246c2b2d3da350f97c0916efe) @potiuk, I hope that they are helpful! I think you have a challenging problem of wanting to scale an interaction that has many expectations around people, and human interactions that require kindness and patience. But there definitely could be some tweaks (e.g., sticking more to traditional expectations of a review, giving details when it's called for and not expecting pre-request expertise, and either minimizing tooling that is unexpected, defaulting to using the testing environment as the base for giving feedback to the contributor, or removing layers of abstraction so it's less confusing). Anyway, I outlined the details much better in that document. Please feel free to share! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #6900: [AIRFLOW-6346] Enhance dag default_view and orientation
milton0825 commented on a change in pull request #6900: [AIRFLOW-6346] Enhance dag default_view and orientation URL: https://github.com/apache/airflow/pull/6900#discussion_r369755076 ## File path: airflow/models/dag.py ## @@ -296,8 +298,16 @@ def __init__( self.max_active_runs = max_active_runs self.dagrun_timeout = dagrun_timeout self.sla_miss_callback = sla_miss_callback -self._default_view = default_view Review comment: no idea either... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6557) Warn if new fields are added to BaseOperator (for serialization)
[ https://issues.apache.org/jira/browse/AIRFLOW-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021470#comment-17021470 ] ASF subversion and git services commented on AIRFLOW-6557: -- Commit 0ea1af0a6c0877dc33525f958e3793c563d59b37 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0ea1af0 ] fixup! [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162) > Warn if new fields are added to BaseOperator (for serialization) > > > Key: AIRFLOW-6557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6557 > Project: Apache Airflow > Issue Type: Improvement > Components: serialization, tests >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > > There should be an automated tests that tells people what to do in case they > add field in BaseOperator. The test should fail and explain how serialisation > should be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6557) Warn if new fields are added to BaseOperator (for serialization)
[ https://issues.apache.org/jira/browse/AIRFLOW-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021474#comment-17021474 ] ASF subversion and git services commented on AIRFLOW-6557: -- Commit 10e37e2ac78375d7c8abf344a2703eb0ac4856c0 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=10e37e2 ] fixup! fixup! [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162) > Warn if new fields are added to BaseOperator (for serialization) > > > Key: AIRFLOW-6557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6557 > Project: Apache Airflow > Issue Type: Improvement > Components: serialization, tests >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > > There should be an automated tests that tells people what to do in case they > add field in BaseOperator. The test should fail and explain how serialisation > should be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-577373099 okay, I've taken a look at the docker-compose approach. Specifically, I've created a `integration-singularity.yml` file that uses a container base that provides Singularity (v3.5.1). ```yaml version: "2.2" services: singularity: image: quay.io/singularity/singularity:v3.5.1 command: -F /etc/alpine-release entrypoint: /usr/bin/tail privileged: true volumes: - /dev/urandom:/dev/random # Required to get non-blocking entropy source ``` Note that I'm using privileged, which likely requires the sudo:true parameter for the testing setup. The traditional entrypoint is the Singularity executable, so I have a basic entrypoint/command to just keep the container running (and allow me to shell inside). Even outside of testing (which is fairly complicated and I'm hoping to not need to reproduce it locally) we can test if the basic interaction with Singularity would work inside the container. ```bash docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" up -d ``` Check that everything went up ok... ```bash docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" logs Attaching to docker-compose_singularity_1_7d898ecbdbb0 singularity_1_7d898ecbdbb0 | 3.10.3 ``` And that we continued running... ```bash $ docker-compose --log-level INFO -f "scripts/ci/docker-compose/integration-singularity.yml" ps Name Command State Ports -- docker-compose_singularity_1_7d898ecbdbb0 /usr/bin/tail -F /etc/alpi ... Up ``` Great! And then we can shell inside. We confirm the version of singularity. ```bash $ docker exec -it 0081d687eeb1 bash bash-5.0# which singularity /usr/local/bin/singularity bash-5.0# singularity --version singularity version 3.5.1 ``` The basic thing we would need to do is start instances. We can actually build the SIF binary successfully, and let's do that first, pulling the container: ```bash $ docker exec -it 3e73d2d5b41e bash bash-5.0# which singularity /usr/local/bin/singularity bash-5.0# singularity pull docker://busybox INFO:Converting OCI blobs to SIF format INFO:Starting build... Getting image source signatures Copying blob bdbbaa22dec6 done Copying config b075e856be done Writing manifest to image destination Storing signatures 2020/01/22 20:24:24 info unpack layer: sha256:bdbbaa22dec6b7fe23106d2c1b1f43d9598cd8fc33706cc27c1d938ecd5bffc7 INFO:Creating SIF file... INFO:Build complete: busybox_latest.sif bash-5.0# ``` Now let's start the instance. ```bash bash-5.0# singularity instance start busybox_latest.sif busybox ERROR: container cleanup failed: no instance found with name busybox FATAL: container creation failed: mount /etc/localtime->/etc/localtime error: while mounting /etc/localtime: mount source /etc/localtime doesn't exist FATAL: failed to start instance: while running /usr/local/libexec/singularity/bin/starter: exit status 255 ``` okay, let's trace back and add this to our docker-compose yml file, we will mount as a volume from the host and redo the above. ```yaml version: "2.2" services: singularity: image: quay.io/singularity/singularity:v3.5.1 command: -F /etc/alpine-release entrypoint: /usr/bin/tail privileged: true volumes: - /etc/localtime:/etc/localtime ``` Zooming ahead, let's recreate and then start the instance: ```bash bash-5.0# singularity instance start busybox_latest.sif busybox INFO:instance started successfully ``` Success! ```bash bash-5.0# singularity instance list INSTANCE NAMEPID IP IMAGE busybox 78 /go/busybox_latest.sif ``` And we can exec to it: ```bash bash-5.0# singularity exec instance://busybox echo "HELLO" HELLO ``` So, although I'm hesitant that something about travis won't provide us with all the permissions that we need, I'm going to give it a try to add a docker compose file to reproduce this. My question for you is how I should go about providing this integration - the examples given previously expose services, and for this case we would want an environment with airflow and singularity in the same space. Do you have thoughts or examples for how to approach this? And then given that we can add a custom container, it would need to use one of the Singularity bases (note they are alpine), and
[GitHub] [airflow] m1racoli commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
m1racoli commented on a change in pull request #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#discussion_r369808495 ## File path: docs/concepts.rst ## @@ -1078,15 +1078,14 @@ For example, consider the following DAG: trigger_rule=TriggerRule.ALL_DONE) task4.set_upstream([task1, task2]) -In the case of this DAG, the ``latest_only`` task will show up as skipped -for all runs except the latest run. ``task1`` is directly downstream of -``latest_only`` and will also skip for all runs except the latest. +In the case of this DAG, the task ``task1`` is directly downstream of +``latest_only`` and will be skipped for all runs except the latest. ``task2`` is entirely independent of ``latest_only`` and will run in all scheduled periods. ``task3`` is downstream of ``task1`` and ``task2`` and because of the default ``trigger_rule`` being ``all_success`` will receive a cascaded skip from ``task1``. ``task4`` is downstream of ``task1`` and -``task2``. It will be first skipped directly by ``LatestOnlyOperator``, -even its ``trigger_rule`` is set to ``all_done``. +``task2``, but it will not be skipped, since its ``trigger_rule`` is set to +``all_done``. Review comment: Good point and thanks! Will update Updating.md. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#discussion_r369817165 ## File path: airflow/kubernetes/pod_generator.py ## @@ -301,51 +352,151 @@ def reconcile_pods(base_pod: k8s.V1Pod, client_pod: k8s.V1Pod) -> k8s.V1Pod: :type client_pod: k8s.V1Pod :return: the merged pods -This can't be done recursively as certain fields are preserved, -some overwritten, and some concatenated, e.g. The command -should be preserved from base, the volumes appended to and -the other fields overwritten. +This can't be done recursively as certain fields some overwritten, and some concatenated. """ +if client_pod is None: +return base_pod + client_pod_cp = copy.deepcopy(client_pod) def merge_objects(base_obj, client_obj): +if not base_obj: +return client_obj +if not client_obj: +return base_obj + +client_obj_cp = copy.deepcopy(client_obj) + for base_key in base_obj.to_dict().keys(): base_val = getattr(base_obj, base_key, None) if not getattr(client_obj, base_key, None) and base_val: -setattr(client_obj, base_key, base_val) +setattr(client_obj_cp, base_key, base_val) +return client_obj_cp def extend_object_field(base_obj, client_obj, field_name): +client_obj_cp = copy.deepcopy(client_obj) base_obj_field = getattr(base_obj, field_name, None) client_obj_field = getattr(client_obj, field_name, None) if not base_obj_field: -return +return client_obj_cp if not client_obj_field: -setattr(client_obj, field_name, base_obj_field) -return +setattr(client_obj_cp, field_name, base_obj_field) +return client_obj_cp appended_fields = base_obj_field + client_obj_field -setattr(client_obj, field_name, appended_fields) - -# Values at the pod and metadata should be overwritten where they exist, -# but certain values at the spec and container level must be conserved. -base_container = base_pod.spec.containers[0] -client_container = client_pod_cp.spec.containers[0] - -extend_object_field(base_container, client_container, 'volume_mounts') -extend_object_field(base_container, client_container, 'env') -extend_object_field(base_container, client_container, 'env_from') -extend_object_field(base_container, client_container, 'ports') -extend_object_field(base_container, client_container, 'volume_devices') -client_container.command = base_container.command -client_container.args = base_container.args -merge_objects(base_pod.spec.containers[0], client_pod_cp.spec.containers[0]) -# Just append any additional containers from the base pod -client_pod_cp.spec.containers.extend(base_pod.spec.containers[1:]) - -merge_objects(base_pod.metadata, client_pod_cp.metadata) - -extend_object_field(base_pod.spec, client_pod_cp.spec, 'volumes') -merge_objects(base_pod.spec, client_pod_cp.spec) -merge_objects(base_pod, client_pod_cp) +setattr(client_obj_cp, field_name, appended_fields) +return client_obj_cp + +if base_pod.spec and not client_pod.spec: +client_pod_cp.spec = base_pod.spec +elif client_pod_cp.spec and base_pod.spec: +client_container = client_pod_cp.spec.containers[0] +base_container = base_pod.spec.containers[0] +cc1 = extend_object_field(base_container, client_container, 'volume_mounts') +cc2 = extend_object_field(base_container, cc1, 'env') +cc3 = extend_object_field(base_container, cc2, 'env_from') +cc4 = extend_object_field(base_container, cc3, 'ports') +cc5 = extend_object_field(base_container, cc4, 'volume_devices') + +cc6 = merge_objects(base_container, cc5) +client_pod_cp.spec.containers[0] = cc6 +# Just append any additional containers from the base pod +client_pod_cp.spec.containers.extend(base_pod.spec.containers[1:]) +merged_spec = extend_object_field(base_pod.spec, client_pod_cp.spec, 'volumes') +client_pod_cp.spec = merge_objects(base_pod.spec, merged_spec) + +client_pod_cp.metadata = merge_objects(base_pod.metadata, client_pod_cp.metadata) +client_pod_cp = merge_objects(base_pod, client_pod_cp) return client_pod_cp + +@staticmethod +def construct_pod( +dag_id: str, +task_id: str, +pod_id: str, +try_number: in
[GitHub] [airflow] m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#issuecomment-577404876 Comments for updating have been added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=h1) Report > Merging [#6075](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/34ed5debc806bc348f82e57359afa01af9ed12ce?src=pr&el=desc) will **decrease** coverage by `0.13%`. > The diff coverage is `16.66%`. [](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6075 +/- ## == - Coverage 85.18% 85.04% -0.14% == Files 710 791 +81 Lines 3947940138 +659 == + Hits3363034136 +506 - Misses 5849 6002 +153 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/hooks/athena.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9hdGhlbmEucHk=) | `60.81% <16.66%> (-6.38%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...rflow/contrib/sensors/sagemaker\_training\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYWluaW5nX3NlbnNvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [244 more](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=footer). Last update [34ed5de...92b753e](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
codecov-io edited a comment on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#issuecomment-571816542 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=h1) Report > Merging [#5970](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c890d066965aa9dbf3016f41cfae45e9a084478a?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#5970 +/- ## == + Coverage 85.34% 85.34% +<.01% == Files 791 791 Lines 4012840127 -1 == + Hits3424734248 +1 + Misses 5881 5879 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/airflow/pull/5970/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=) | `100% <100%> (+10%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=footer). Last update [c890d06...eba7aaf](https://codecov.io/gh/apache/airflow/pull/5970?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6211: [AIRFLOW-5569] Improve Pod Launcher Logging
stale[bot] commented on issue #6211: [AIRFLOW-5569] Improve Pod Launcher Logging URL: https://github.com/apache/airflow/pull/6211#issuecomment-577434319 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=h1) Report > Merging [#6954](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3e2817c3e967a102c166dffd909a5cc030271e25?src=pr&el=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `93.65%`. [](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6954 +/- ## == - Coverage 84.96% 84.64% -0.33% == Files 683 791 +108 Lines 3916440137 +973 == + Hits3327633973 +697 - Misses 5888 6164 +276 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/macros/hive.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9tYWNyb3MvaGl2ZS5weQ==) | `38.7% <ø> (ø)` | :arrow_up: | | [airflow/operators/gcs\_to\_bq.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2JxLnB5) | `70.58% <ø> (ø)` | :arrow_up: | | [airflow/operators/cassandra\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvY2Fzc2FuZHJhX3RvX2djcy5weQ==) | `64.91% <ø> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `95.13% <ø> (ø)` | | | [airflow/gcp/example\_dags/example\_dataflow.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YWZsb3cucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `52.94% <ø> (-32.36%)` | :arrow_down: | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/python\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uX29wZXJhdG9yLnB5) | `100% <ø> (+4.19%)` | :arrow_up: | | ... and [455 more](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=footer). Last update [3e2817c...50a2bed](https://codecov.io/gh/apache/airflow/pull/6954?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
