[GitHub] [airflow] OmerJog commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators to contrib

[GitBox]

OmerJog commented on issue #6824: [AIRFLOW-6258] add CloudFormation operators 
to contrib
URL: https://github.com/apache/airflow/pull/6824#issuecomment-566428846
 
 
   @aviemzur can you please fix errors?
   
   ```
   airflow/contrib/operators/cloudformation_create_stack_operator.py:1:0: 
C0111: Missing module docstring (missing-docstring)
   airflow/contrib/operators/cloudformation_create_stack_operator.py:52:8: 
C0103: Attribute name "StackName" doesn't conform to snake_case naming style 
(invalid-name)
   airflow/contrib/operators/cloudformation_create_stack_operator.py:53:8: 
C0103: Attribute name "TimeoutInMinutes" doesn't conform to snake_case naming 
style (invalid-name)
   airflow/contrib/operators/cloudformation_create_stack_operator.py:49:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   airflow/contrib/operators/cloudformation_create_stack_operator.py:49:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   airflow/contrib/operators/cloudformation_create_stack_operator.py:49:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   * Module 
airflow.contrib.operators.cloudformation_delete_stack_operator
   airflow/contrib/operators/cloudformation_delete_stack_operator.py:1:0: 
C0111: Missing module docstring (missing-docstring)
   airflow/contrib/operators/cloudformation_delete_stack_operator.py:48:8: 
C0103: Attribute name "StackName" doesn't conform to snake_case naming style 
(invalid-name)
   airflow/contrib/operators/cloudformation_delete_stack_operator.py:45:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   airflow/contrib/operators/cloudformation_delete_stack_operator.py:45:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   airflow/contrib/operators/cloudformation_delete_stack_operator.py:45:57: 
E1101: Method '__init__' has no '__wrapped__' member (no-member)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] OmerJog commented on issue #6743: fix Duplicate entry 'xxx' for key 'dag_id' errors when trigger multip…

[GitBox]

OmerJog commented on issue #6743: fix Duplicate entry 'xxx' for key 'dag_id' 
errors when trigger multip…
URL: https://github.com/apache/airflow/pull/6743#issuecomment-566429246
 
 
   @YouZhengChuan This is a code change it needs a Jira ticket


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-918) Improve bulk_load function for MySqlHook

[jack (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997977#comment-16997977
 ] 

jack commented on AIRFLOW-918:
--

[~ash] can be closed as duplicate of 
https://issues.apache.org/jira/browse/AIRFLOW-5921

Also this is a child of https://issues.apache.org/jira/browse/AIRFLOW-3886

> Improve bulk_load function for MySqlHook
> 
>
> Key: AIRFLOW-918
> URL: https://issues.apache.org/jira/browse/AIRFLOW-918
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.7.1.3
>Reporter: Ali Uz
>Priority: Minor
>  Labels: easyfix, patch
>
> I think we can improve the `bulk_load` function in MySqlHook by adding a few 
> more parameters. For example, if I want to run a LOAD DATA command like the 
> following:
> ```
> LOAD DATA LOCAL INFILE 'abc.csv' INTO TABLE abc
> FIELDS TERMINATED BY ',' 
> ENCLOSED BY '"' 
> LINES TERMINATED BY '\r\n'
> IGNORE 1 LINES
> ```
> I would expect to supply the delimiter parameters, enclosing quotes 
> parameter, line terminating parameter and ignore line number parameter.
> The current function only applies the following command:
> ```
> LOAD DATA LOCAL INFILE 'abc.csv' INTO TABLE abc
> ```
> It would be great if we could extend it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] baolsen commented on issue #6773: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

baolsen commented on issue #6773: [AIRFLOW-6038] AWS DataSync example_dags added
URL: https://github.com/apache/airflow/pull/6773#issuecomment-566441433
 
 
   Hey @potiuk , friendly reminder to review when you can. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator

[GitBox]

codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in 
DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#issuecomment-556095248
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=h1) 
Report
   > Merging 
[#6552](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9ed4895263f8683a576c3fb332598c5efb26284b?src=pr&el=desc)
 will **increase** coverage by `0.54%`.
   > The diff coverage is `84.61%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6552/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6552  +/-   ##
   ==
   + Coverage   83.77%   84.32%   +0.54% 
   ==
 Files 669  676   +7 
 Lines   3755238363 +811 
   ==
   + Hits3146032348 +888 
   + Misses   6092 6015  -77
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `90.9% <84.61%> (-5.76%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_record\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3JlY29yZF9zZW5zb3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_table\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3RhYmxlX3NlbnNvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/jira\_hook.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2ppcmFfaG9vay5weQ==)
 | `0% <0%> (-80%)` | :arrow_down: |
   | 
[airflow/contrib/operators/jira\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9qaXJhX29wZXJhdG9yLnB5)
 | `0% <0%> (-76.67%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/jira\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvamlyYV9zZW5zb3IucHk=)
 | `0% <0%> (-60%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `66.66% <0%> (-33.34%)` | :arrow_down: |
   | ... and [152 
more](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=footer). 
Last update 
[9ed4895...2e77089](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator

[GitBox]

codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in 
DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#issuecomment-556095248
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=h1) 
Report
   > Merging 
[#6552](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9ed4895263f8683a576c3fb332598c5efb26284b?src=pr&el=desc)
 will **increase** coverage by `0.54%`.
   > The diff coverage is `84.61%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6552/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6552  +/-   ##
   ==
   + Coverage   83.77%   84.32%   +0.54% 
   ==
 Files 669  676   +7 
 Lines   3755238363 +811 
   ==
   + Hits3146032348 +888 
   + Misses   6092 6015  -77
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `90.9% <84.61%> (-5.76%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_record\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3JlY29yZF9zZW5zb3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_table\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3RhYmxlX3NlbnNvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/jira\_hook.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2ppcmFfaG9vay5weQ==)
 | `0% <0%> (-80%)` | :arrow_down: |
   | 
[airflow/contrib/operators/jira\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9qaXJhX29wZXJhdG9yLnB5)
 | `0% <0%> (-76.67%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/jira\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvamlyYV9zZW5zb3IucHk=)
 | `0% <0%> (-60%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `66.66% <0%> (-33.34%)` | :arrow_down: |
   | ... and [152 
more](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=footer). 
Last update 
[9ed4895...2e77089](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in DockerSwarmOperator

[GitBox]

codecov-io edited a comment on issue #6552: AIRFLOW-5850: Capture task logs in 
DockerSwarmOperator
URL: https://github.com/apache/airflow/pull/6552#issuecomment-556095248
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=h1) 
Report
   > Merging 
[#6552](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9ed4895263f8683a576c3fb332598c5efb26284b?src=pr&el=desc)
 will **increase** coverage by `0.54%`.
   > The diff coverage is `84.61%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6552/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6552  +/-   ##
   ==
   + Coverage   83.77%   84.32%   +0.54% 
   ==
 Files 669  676   +7 
 Lines   3755238363 +811 
   ==
   + Hits3146032348 +888 
   + Misses   6092 6015  -77
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=)
 | `90.9% <84.61%> (-5.76%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_record\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3JlY29yZF9zZW5zb3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/cassandra\_table\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2Fzc2FuZHJhX3RhYmxlX3NlbnNvci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/jira\_hook.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2ppcmFfaG9vay5weQ==)
 | `0% <0%> (-80%)` | :arrow_down: |
   | 
[airflow/contrib/operators/jira\_operator.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9qaXJhX29wZXJhdG9yLnB5)
 | `0% <0%> (-76.67%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/jira\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvamlyYV9zZW5zb3IucHk=)
 | `0% <0%> (-60%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5)
 | `66.66% <0%> (-33.34%)` | :arrow_down: |
   | ... and [152 
more](https://codecov.io/gh/apache/airflow/pull/6552/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=footer). 
Last update 
[9ed4895...2e77089](https://codecov.io/gh/apache/airflow/pull/6552?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6670: [AIRFLOW-4816]MySqlToS3Operator

[GitBox]

nuclearpinguin commented on a change in pull request #6670: 
[AIRFLOW-4816]MySqlToS3Operator
URL: https://github.com/apache/airflow/pull/6670#discussion_r358683309
 
 

 ##
 File path: airflow/operators/mysql_to_s3_operator.py
 ##
 @@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfer data from MySQL into a S3 bucket
+"""
+from io import StringIO
+from typing import Optional, Union
+
+import numpy as np
+import pandas as pd
+
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.utils.decorators import apply_defaults
+
+
+class MySQLToS3Operator(BaseOperator):
+"""
+Saves data from an specific MySQL query into a file in S3.
+
+:param query: the sql query to be executed.
+:type query: str
+:param s3_bucket: bucket where the data will be stored
+:type s3_bucket: str
+:param s3_key: desired key for the file. It includes the name of the file.
+If a csv file is wanted, the param must end with ".csv".
+:type s3_key: str
+:param mysql_conn_id: reference to a specific mysql database
+:type mysql_conn_id: str
+ :param aws_conn_id: reference to a specific S3 connection
+:type aws_conn_id: str
+:param verify: Whether or not to verify SSL certificates for S3 connection.
+By default SSL certificates are verified.
+You can provide the following values:
+- False: do not validate SSL certificates. SSL will still be used
 
 Review comment:
   ```suggestion
   
   - False: do not validate SSL certificates. SSL will still be used
   ```
   I think there should be a new line. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6026) Use ccntextlib to redirect stderr and stdout

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-6026.

Fix Version/s: (was: 1.10.7)
   2.0.0
   Resolution: Fixed

> Use ccntextlib to redirect stderr and stdout
> 
>
> Key: AIRFLOW-6026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6026
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (AIRFLOW-6026) Use ccntextlib to redirect stderr and stdout

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor reopened AIRFLOW-6026:


[~kamil.bregula] please only mark issues as Resolved, not Closed. (We can't 
edit anything about a Closed issue)

Additionally this doesn't work for Py2 so should have been targeted at 2.0.0 
not 1.10.7

> Use ccntextlib to redirect stderr and stdout
> 
>
> Key: AIRFLOW-6026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6026
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 1.10.7
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6035) Remove comand method in TaskInstance

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-6035:
---
Fix Version/s: (was: 1.10.7)
   2.0.0

> Remove comand method in TaskInstance
> 
>
> Key: AIRFLOW-6035
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6035
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Trivial
> Fix For: 2.0.0
>
>
> This method is not used. In addition, this method does not work properly 
> because the arguments should be processed using the shlex.quote function.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5417) Webserver won't start with long DagBag parsing

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998062#comment-16998062
 ] 

ASF subversion and git services commented on AIRFLOW-5417:
--

Commit 54fa91d6a61cb910d1e0aa39ddb58c9dc7b4e48d in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=54fa91d ]

[AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)


> Webserver won't start with long DagBag parsing
> --
>
> Key: AIRFLOW-5417
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5417
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Affects Versions: 1.10.5
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Major
> Fix For: 2.0.0, 1.10.7
>
>
> When DB disconnect during DagBag creation, webserver will fail to start. This 
> becomes a big issue when DagBag is large enough that creating it takes more 
> time than DB server side connection expire limit. As it will fail webserver 
> start up determinisitically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5730) Enable get_pandas_df on PinotDbApiHook

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998061#comment-16998061
 ] 

ASF subversion and git services commented on AIRFLOW-5730:
--

Commit a3253b1fa99411612ed008b81a7d01f3c9486c42 in airflow's branch 
refs/heads/v1-10-test from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a3253b1 ]

[AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)


(cherry picked from commit 8f1a585b58e6d8091f4524e6cfb09c606e828825)


> Enable get_pandas_df on PinotDbApiHook
> --
>
> Key: AIRFLOW-5730
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5730
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.5
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 2.0.0, 1.10.7
>
>
> Currently, DruidDbApiHook and PinotDbApiHook disable their {{get_pandas_df}} 
> methods by raising {{NotImplementedError}}.
> But they actually work as inherited from DbApiHook, as follows:
> {code}
> $ git diff
> diff --git a/airflow/contrib/hooks/pinot_hook.py 
> b/airflow/contrib/hooks/pinot_hook.py
> index e617f8e9b..0864b3584 100644
> --- a/airflow/contrib/hooks/pinot_hook.py
> +++ b/airflow/contrib/hooks/pinot_hook.py
> @@ -90,8 +90,5 @@ class PinotDbApiHook(DbApiHook):
>  def set_autocommit(self, conn, autocommit):
>  raise NotImplementedError()
>  
> -def get_pandas_df(self, sql, parameters=None):
> -raise NotImplementedError()
> -
>  def insert_rows(self, table, rows, target_fields=None, 
> commit_every=1000):
>  raise NotImplementedError()
> diff --git a/airflow/hooks/druid_hook.py b/airflow/hooks/druid_hook.py
> index c3cd3cd71..e2e20f1ec 100644
> --- a/airflow/hooks/druid_hook.py
> +++ b/airflow/hooks/druid_hook.py
> @@ -158,8 +158,5 @@ class DruidDbApiHook(DbApiHook):
>  def set_autocommit(self, conn, autocommit):
>  raise NotImplementedError()
>  
> -def get_pandas_df(self, sql, parameters=None):
> -raise NotImplementedError()
> -
>  def insert_rows(self, table, rows, target_fields=None, 
> commit_every=1000):
>  raise NotImplementedError()
> {code}
> {code:title=Druid example}
> $ airflow connections list
> (snip)
> ├┼─┼───┼┼┼──┼┤
> │ 'druid_broker_default' │ 'druid-broker'  │ 'localhost'  
>  │ 8082   │ False  │ True │ 
> 'gABdrxvt...M1ideRO8233QG' │
> ╘╧═╧═══╧╧╧══╧╛
> $ ipython
> (snip)
> In [2]: from airflow.hooks.druid_hook import DruidDbApiHook   
>   
>  
> In [3]: DruidDbApiHook().get_pandas_df("SELECT * FROM wikipedia WHERE 
> sum_delta > %(num)d", {"num": 2000})  
>  
> [2019-10-23 23:28:18,606] {base_hook.py:89} INFO - Using connection to: id: 
> druid_broker_default. Host: localhost, Port: 8082, Schema: None, Login: None, 
> Password: None, extra: {'schema': 'http', 'endpoint': '/druid/v2/sql'}
> [2019-10-23 23:28:18,607] {druid_hook.py:140} INFO - Get the connection to 
> druid broker on localhost using user None
> Out[3]: 
>__timechannel cityName 
>comment  ...  sum_deleted sum_delta sum_metroCode  
>   user
> 02015-09-12T00:00:00.000Z  #en.wikipedia   Archiving case from 
> [[Wikipedia:Sockpuppet inv...  ...0  3360 0   
> Bbb23
> 12015-09-12T00:00:00.000Z  #ja.wikipedia   
> [[Special:Contributions/119.224.209.170|119.22...  ...0  6853 
> 0 Kkairri
> 22015-09-12T01:00:00.000Z  #en.wikipedia  
>/* Hong Kong */  ...0  4500 0  
>Bertaut
> 32015-09-12T01:00:00.000Z  #en.wikipedia   Archiving 1 
> discussion(s) from [[User talk:New...  ...0  3599 
> 0  Lowercase sigmabot III
> 42015-09-12T01:00:00.000Z  #en.wikipedia   [[WP:AES|←]]Created 
> page with '{{Infobox wildf...  ...0 13335 0   
>Orygun
> ........  ... 
>...  ...  ...   ...   

[jira] [Commented] (AIRFLOW-5925) Upgrade funcsigs dependency

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998060#comment-16998060
 ] 

ASF subversion and git services commented on AIRFLOW-5925:
--

Commit 8ce93d627d0f3ac122a5f7fbfb78ca4a6225d18e in airflow's branch 
refs/heads/v1-10-test from aoen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8ce93d6 ]

[AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)


> Upgrade funcsigs dependency
> ---
>
> Key: AIRFLOW-5925
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5925
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Minor
> Fix For: 2.0.0, 1.10.7
>
>
> Loosen funcsigs version range restrictions to allow it to be upgraded.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6773#discussion_r358705820
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/datasync.py
 ##
 @@ -18,34 +16,48 @@
 # under the License.
 
 """
-Get, Create, Update, Delete and execute an AWS DataSync Task.
+Create, get, update, execute and delete an AWS DataSync Task.
 """
 
+import random
+
 from airflow.exceptions import AirflowException
 from airflow.models import BaseOperator
 from airflow.providers.amazon.aws.hooks.datasync import AWSDataSyncHook
 from airflow.utils.decorators import apply_defaults
 
 
-class AWSDataSyncCreateTaskOperator(BaseOperator):
-r"""Create an AWS DataSync Task.
+# pylint: disable=too-many-instance-attributes, too-many-arguments
+class AWSDataSyncOperator(BaseOperator):
+r"""Find, Create, Update, Execute and Delete AWS DataSync Tasks.
 
-If there are existing Locations which match the specified
-source and destination URIs then these will be used for the Task.
-Otherwise, new Locations can be created automatically,
-depending on input parameters.
+If ``do_xcom_push`` is True, then the DataSync TaskArn and TaskExecutionArn
+which were executed will be pushed to an XCom.
 
-If ``do_xcom_push`` is True, the TaskArn which is created
-will be pushed to an XCom.
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AWSDataSyncOperator`
+
+.. note:: There may be 0, 1, or many existing DataSync Tasks defined in 
your AWS
+environment. The default behavior is to create a new Task if there are 
0, or
+execute the Task if there was 1 Task, or fail if there were many Tasks.
 
 :param str aws_conn_id: AWS connection to use.
-:param str source_location_uri: Source location URI.
+:param int wait_interval_seconds: Time to wait between two
+consecutive calls to check TaskExecution status.
+:param str task_arn: AWS DataSync TaskArn to use. If None, then this 
operator will
+attempt to either search for an existing Task or attempt to create a 
new Task.
+:param str source_location_uri: Source location URI to search for. All 
DataSync
+Tasks with a LocationArn with this URI will be considered.
 Example: ``smb://server/subdir``
-:param str destination_location_uri: Destination location URI.
+:param str destination_location_uri: Destination location URI to search 
for.
+All DataSync Tasks with a LocationArn with this URI will be considered.
 Example: ``s3://airflow_bucket/stuff``
-:param bool case_sensitive_location_search: Whether or not to do a
-case-sensitive search for each Location URI.
-:param dict create_task_kwargs: If no suitable TaskArn is found,
+:param str choose_task_strategy: If multiple Tasks match, one must be 
chosen to
 
 Review comment:
   I think it would be a bit better to have a bool here 
"allow_random_task_selection" and "allow_random_location_selection" or similar. 
I think it is a bit overkill to have 'random' string as the only strategy 
implemented. My thought when I was reviewing it was "ok - what are the other 
strategies ?". 
   
   I believe you had in mind to make it future-compatible. In the future we 
might be able to not change the interface and use other strategies, but I think 
this is a bit of a trap (that's what is suggested by the documentation). I 
think this is a bit of a trap - I prefer to implement *just* what is needed in 
the simplest possible way. We can always add new parameters in the future in 
case we find out that we need. I think the initial idea with callable was 
pretty good for the more complex cases (much better than imaginary 
'choose_first', 'choose_first_alphabetically', 'fifo', 'lifo' etc. etc.). In 
this case it seems that the "random" bool might simply survive as bool flag (if 
callable is not specified). In such case str 'random' strategy will be 
sttrange. 
   
   If we want to introduce 'str' strategy in the future we might always add it 
as a new field that can even be working together with random flag (for example 
the strategy might pre-select subset of tasks and THEN random choice will be 
used).
   
   I do not have a very strong opinion here - I leave it up to you to either 
fix or leave as it is, but at least consider this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6773#discussion_r358712118
 
 

 ##
 File path: tests/providers/amazon/aws/operators/test_datasync.py
 ##
 @@ -415,29 +464,42 @@ def test_get_many_tasks(self, mock_get_conn):
 
 self.client.create_task(
 SourceLocationArn=self.source_location_arn,
-DestinationLocationArn=self.destination_location_arn
+DestinationLocationArn=self.destination_location_arn,
 )
 
 # Check how many tasks and locations we have
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 2)
+self.assertEqual(len(tasks["Tasks"]), 2)
 locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 2)
+self.assertEqual(len(locations["Locations"]), 2)
 
 # Execute the task
-result = self.datasync.execute(None)
-self.assertIsNotNone(result)
-
-task_arns = result
-self.assertIsNotNone(task_arns)
-self.assertTrue(task_arns)
-self.assertEqual(len(task_arns), 2)
+with self.assertRaises(AirflowException):
+self.datasync.execute(None)
 
 # Assert 0 additional task and 0 additional locations
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 2)
+self.assertEqual(len(tasks["Tasks"]), 2)
 
 Review comment:
   NIT: should we split that to two tests?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6773: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6773#discussion_r358714148
 
 

 ##
 File path: tests/providers/amazon/aws/operators/test_datasync.py
 ##
 @@ -197,88 +213,107 @@ def test_create_task(self, mock_get_conn):
 self.set_up_operator()
 # Delete all tasks:
 tasks = self.client.list_tasks()
-for task in tasks['Tasks']:
-self.client.delete_task(TaskArn=task['TaskArn'])
+for task in tasks["Tasks"]:
+self.client.delete_task(TaskArn=task["TaskArn"])
 
 # Check how many tasks and locations we have
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 0)
+self.assertEqual(len(tasks["Tasks"]), 0)
 locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 2)
+self.assertEqual(len(locations["Locations"]), 2)
 
 # Execute the task
 result = self.datasync.execute(None)
 self.assertIsNotNone(result)
-task_arn = result
+task_arn = result["TaskArn"]
 
 # Assert 1 additional task and 0 additional locations
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 1)
+self.assertEqual(len(tasks["Tasks"]), 1)
 locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 2)
+self.assertEqual(len(locations["Locations"]), 2)
 
 # Check task metadata
 task = self.client.describe_task(TaskArn=task_arn)
-self.assertEqual(task['Options'], CREATE_TASK_KWARGS['Options'])
+self.assertEqual(task["Options"], CREATE_TASK_KWARGS["Options"])
 
-def test_create_task_even_if_one_exists(self, mock_get_conn):
+def test_create_task_and_location(self, mock_get_conn):
 # ### Set up mocks:
 mock_get_conn.return_value = self.client
 # ### Begin tests:
 
 self.set_up_operator()
+# Delete all tasks:
+tasks = self.client.list_tasks()
+for task in tasks["Tasks"]:
+self.client.delete_task(TaskArn=task["TaskArn"])
+# Delete all locations:
+locations = self.client.list_locations()
+for location in locations["Locations"]:
+self.client.delete_location(LocationArn=location["LocationArn"])
 
 # Check how many tasks and locations we have
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 1)
+self.assertEqual(len(tasks["Tasks"]), 0)
 locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 2)
+self.assertEqual(len(locations["Locations"]), 0)
 
 # Execute the task
 result = self.datasync.execute(None)
 self.assertIsNotNone(result)
-task_arn = result
 
-# Assert 1 additional task and 0 additional locations
+# Assert 1 additional task and 2 additional locations
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 2)
+self.assertEqual(len(tasks["Tasks"]), 1)
 locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 2)
+self.assertEqual(len(locations["Locations"]), 2)
 
-# Check task metadata
-task = self.client.describe_task(TaskArn=task_arn)
-self.assertEqual(task['Options'], CREATE_TASK_KWARGS['Options'])
-
-def test_create_task_and_location(self, mock_get_conn):
+def test_dont_create_task(self, mock_get_conn):
 # ### Set up mocks:
 mock_get_conn.return_value = self.client
 # ### Begin tests:
 
-self.set_up_operator()
-# Delete all tasks:
 tasks = self.client.list_tasks()
-for task in tasks['Tasks']:
-self.client.delete_task(TaskArn=task['TaskArn'])
-# Delete all locations:
-locations = self.client.list_locations()
-for location in locations['Locations']:
-self.client.delete_location(LocationArn=location['LocationArn'])
+tasks_before = len(tasks["Tasks"])
+
+self.set_up_operator(task_arn=self.task_arn)
+self.datasync.execute(None)
 
-# Check how many tasks and locations we have
 tasks = self.client.list_tasks()
-self.assertEqual(len(tasks['Tasks']), 0)
-locations = self.client.list_locations()
-self.assertEqual(len(locations['Locations']), 0)
+tasks_after = len(tasks["Tasks"])
+self.assertEqual(tasks_before, tasks_after)
 
-# Execute the task
+def create_task_many_locations(self, mock_get_conn):
+# ### Set up mocks:
+mock_get_conn.return_value = self.client
+# ### Begin tests:
+
+# Create duplicate source location to choose from
+self.client.create_location_smb(
+**MOCK_DATA[

[jira] [Commented] (AIRFLOW-5345) Extract SqlSensor's get_hook functionality to an auxiliary function

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998064#comment-16998064
 ] 

ASF subversion and git services commented on AIRFLOW-5345:
--

Commit 424c83bddcdefa4c0913f05bb72315ef1b5e2b7c in airflow's branch 
refs/heads/v1-10-test from Guy Maliar
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=424c83b ]

[AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)

Due to security reasons we have an alternative PostgreSQL hook that fetches 
credentials dynamically.
We would still like to use the `SqlSensor` functionality as is as our hook is 
still backed by sqlalchemy.
By extracting SqlSensor's `connection.get_hook` into a method we can override 
the `_get_hook` method without affecting the functionality of the hook.

(cherry picked from commit a8cb800662d674369d17d773fcfd3fcfac6bb2f5)


> Extract SqlSensor's get_hook functionality to an auxiliary function
> ---
>
> Key: AIRFLOW-5345
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5345
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.4
>Reporter: Guy
>Assignee: Guy
>Priority: Minor
> Fix For: 1.10.7
>
>
> Due to security reasons we have an alternative PostgreSQL hook that fetches 
> credentials dynamically.
> We would still like to use the SqlSensor functionality as is as our hook is 
> still backed by sqlalchemy.
> By extracting SqlSensor's get_hook into a method we can override the 
> _get_hook method without affecting the functionality of the hook.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5463) Variable set is not atomic

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998068#comment-16998068
 ] 

ASF subversion and git services commented on AIRFLOW-5463:
--

Commit aceb61283c810bd2999af2889ac9937f83ccf226 in airflow's branch 
refs/heads/v1-10-test from Robert Young
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=aceb612 ]

[AIRFLOW-5463] Use same session to delete and add variable in set (#6807)

Why:
* In our system we had a postgres connection error during Variable.set
resulting in the variable being deleted. The intention of this change is
that an error should leave the variable unchanged.

(cherry picked from commit 5483ae4992f888ac467a9cf4977a01331fb10160)


> Variable set is not atomic
> --
>
> Key: AIRFLOW-5463
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5463
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core, models
>Affects Versions: 1.10.5
>Reporter: Cedrik Neumann
>Priority: Major
> Fix For: 1.10.7
>
>
> The function \{{Variable.set}} deletes the variable first
> [https://github.com/apache/airflow/blob/1.10.5/airflow/models/variable.py#L137]
> although it doesn't pass the DB session as an argument, thus delete and add 
> don't run in an atomic operation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4940) DynamoDB to S3 backup operator

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998069#comment-16998069
 ] 

ASF subversion and git services commented on AIRFLOW-4940:
--

Commit d2a78162c756a0006c4f856a96a172e3ecad8aeb in airflow's branch 
refs/heads/v1-10-test from Chao-Han Tsai
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d2a7816 ]

[AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)

Add an Airflow operator that replicates a
DynamoDB table to S3.

(cherry picked from commit 46cbeea376dfb02bf70fbcf310287733145974dd)


> DynamoDB to S3 backup operator
> --
>
> Key: AIRFLOW-4940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4940
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: aws
>Affects Versions: 1.10.4
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.7
>
>
> Add an Airflow operator that back up DynamoDB table to S3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998063#comment-16998063
 ] 

ASF subversion and git services commented on AIRFLOW-5582:
--

Commit eccef9d68382c5c8493ef24bf12434c1758cd0bf in airflow's branch 
refs/heads/v1-10-test from Scott Liu
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eccef9d ]

[AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)

- add tests
- update docs

Co-Authored-By: Felix Uellendall 
(cherry picked from commit d03fb28e69958d8667483d4dbe083fa097ec08e7)


> AutoCommit in jdbc is missing get_autocommit
> 
>
> Key: AIRFLOW-5582
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5582
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database, hooks
>Affects Versions: 1.10.0
>Reporter: Scott Liu
>Priority: Major
> Fix For: 1.10.7
>
>
> Some of JDBC driver does not support autoCommit=false, such as 
> org.apache.hive.jdbc.HiveDrive.
> JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, 
> DbApiHook.get_autocommit retrieve by conn.autocommit.
> After DbApiHook.execute(sql), conn.commit() will throws exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5082) add subject in aws sns hook

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998066#comment-16998066
 ] 

ASF subversion and git services commented on AIRFLOW-5082:
--

Commit f0537e8b0e9d435c0d00dfadad0dce0476a79f7d in airflow's branch 
refs/heads/v1-10-test from SHAKEEL
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f0537e8 ]

[AIRFLOW-5082] Add subject in AwsSnsHook (#5694)


(cherry picked from commit 919bb8c1cbb36679c02ca3f8890c300e1527c08b)


> add subject in aws sns hook
> ---
>
> Key: AIRFLOW-5082
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5082
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.4
>Reporter: MOHAMMD SHAKEEL SHAIK
>Assignee: MOHAMMD SHAKEEL SHAIK
>Priority: Major
> Fix For: 1.10.7
>
>
> While sending SNS notification to AWS. The subject is an optional field. If 
> we don't send Subject AWS will add default SNS Subject to email "*AWS 
> Notification Message*". If anyone wants to add a different Subject. They can 
> send Subject parameter in AWS SNS hook. 
>  
> It is also optional only



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6263) Fix broken pywinrm integration

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998067#comment-16998067
 ] 

ASF subversion and git services commented on AIRFLOW-6263:
--

Commit 025bf542ad64a436a9c84dd717038244a30d7ab9 in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=025bf54 ]

[AIRFLOW-6263] Fix broken WinRM integration (#6832)


(cherry picked from commit 21ca977c36f4e029d9eeba659875dd7101b035de)


> Fix broken pywinrm integration
> --
>
> Key: AIRFLOW-6263
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6263
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib
>Affects Versions: 1.10.6
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.7
>
>
> WinRMHook & WinRMOperator currently fails with the following error:
> {noformat}
> File 
> "/home/mendesm/airflow/venv/lib/python3.6/site-packages/airflow/contrib/hooks/winrm_hook.py",
>  line 232, in get_conn
> [2019-12-16 13:07:09,489] {base_task_runner.py:115} INFO - Job 16: Subtask 
> wintask1 raise AirflowException(error_msg)
> [2019-12-16 13:07:09,489] {base_task_runner.py:115} INFO - Job 16: Subtask 
> wintask1 airflow.exceptions.AirflowException: Error connecting to host: 
> 192.168.56.1, error: __init__() got an unexpected keyword argument 
> 'message_encryption'
> {noformat}
> This is because 'pywinrm==0.2.2' does not work



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5715) Make email, owner context available

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998065#comment-16998065
 ] 

ASF subversion and git services commented on AIRFLOW-5715:
--

Commit 6e311689e7e2debfc398e542f8975c95d2ef79bf in airflow's branch 
refs/heads/v1-10-test from Tao Feng
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6e31168 ]

[AIRFLOW-5715] Make email, owner context available (#6385)


(cherry picked from commit 641b8aaf04bcf68311cd490481360ea93a3d360d)


> Make email, owner context available
> ---
>
> Key: AIRFLOW-5715
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5715
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.10.5
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
> Fix For: 1.10.7
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #6569: [AIRFLOW-5916] ADDED PASSWORD-ALIAS EXTRA OPTION

[GitBox]

mik-laj commented on issue #6569: [AIRFLOW-5916] ADDED PASSWORD-ALIAS EXTRA 
OPTION
URL: https://github.com/apache/airflow/pull/6569#issuecomment-566483519
 
 
   Can you add tests to avoid regression?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-2227) Allow variable migrations by adding deletion

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-2227.
-
Resolution: Fixed

> Allow variable migrations by adding deletion
> 
>
> Key: AIRFLOW-2227
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2227
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: models
>Reporter: nadav
>Priority: Minor
> Fix For: 2.0.0
>
>
> In order to allow using migration (via alembic or customized scripts) one 
> need the ability to delete variables (delete function).
> Another utility function was added to allow retrieval of all the current 
> variables keys.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-2227) Allow variable migrations by adding deletion

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-2227:

Fix Version/s: (was: 2.0.0)
   1.10.7

> Allow variable migrations by adding deletion
> 
>
> Key: AIRFLOW-2227
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2227
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: models
>Reporter: nadav
>Priority: Minor
> Fix For: 1.10.7
>
>
> In order to allow using migration (via alembic or customized scripts) one 
> need the ability to delete variables (delete function).
> Another utility function was added to allow retrieval of all the current 
> variables keys.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-2227) Allow variable migrations by adding deletion

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998080#comment-16998080
 ] 

ASF subversion and git services commented on AIRFLOW-2227:
--

Commit 255e89f2ea6b0763a7102beda190fc103d677987 in airflow's branch 
refs/heads/v1-10-test from Jordan Sanders
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=255e89f ]

[AIRFLOW-2227] Add delete method to Variable class (#4963)


> Allow variable migrations by adding deletion
> 
>
> Key: AIRFLOW-2227
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2227
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: models
>Reporter: nadav
>Priority: Minor
> Fix For: 1.10.7
>
>
> In order to allow using migration (via alembic or customized scripts) one 
> need the ability to delete variables (delete function).
> Another utility function was added to allow retrieval of all the current 
> variables keys.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5313) Add params support for awsbatch_operator

[ASF subversion and git services (Jira)]

[jira] [Updated] (AIRFLOW-5649) Skipping tests for non relevant changes

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5649:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Skipping tests for non relevant changes
> ---
>
> Key: AIRFLOW-5649
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5649
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>
> We can skip running tests for some changes that are not relevant:
> All the tests will always run for direct pushes. This will run all tests in 
> master branch and v1-10-tests always (no matter if they are cron job or not).
> For PRs it works like that:
>  * if there are no changes in .py files -> the separate pylint check will be 
> skipped altogether (pre-commit will skip the checks anyway in this case but 
> we save time to pull/build images).
>  * if there is no change in .py .js .html . all unit tests will be skipped
>  * if there is a change in .py .js .html files but no kubernetes/* nor 
> tests/kubernetes/* nor airflow/www/.* the Kubernetes unit tests will be 
> skipped
> We still run all test in CRON on master so in case of any false negatives we 
> will catch them anyway.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5886) Copying sources in Dockerfile can be selective

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5886:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Copying sources in Dockerfile can be selective
> --
>
> Key: AIRFLOW-5886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5886
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>
> This change will further improve time of rebuilds for docker image when your 
> sources change (very useful in case of building kubernetes image). It adds 
> only directories that are needed (it is synchronised with .dockerignore and 
> local mounts) and in the sequence that reflects frequency of changes. Also 
> pip install is not done again after sources change (there is no point) so the 
>  build is much faster when only sources or test file change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6001) Lazy load CLI commands

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6001:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Lazy load CLI commands
> --
>
> Key: AIRFLOW-6001
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6001
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>
> Currently, CLI loads all packages even when the command is not used in any 
> way, which takes CPU time and memory.  I suggest that only commands that are 
> executed be loaded. This can speed up task launch time and also improve user 
> productivity.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6000) Fix the test randomly fail - TestCliConnections.test_cli_connections_add_delete

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6000:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Fix the test randomly fail  - 
> TestCliConnections.test_cli_connections_add_delete
> 
>
> Key: AIRFLOW-6000
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6000
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli, tests
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>
> This test requires a clean database to be initialized. There must be no 
> additional connections and there must be default connections. I suggest that 
> the test automatically initialize the database from scratch. In the next 
> steps I would like to introduce full refactoring of CLI tests to increase 
> coverage, but for now I need to increase stability and repeatability to be 
> able to develop another change - optimization of CLI startup time. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5893) Refactor CLI tests

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5893:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Refactor CLI tests
> --
>
> Key: AIRFLOW-5893
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5893
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>
> Hello
> Currently, CLI tests are not ordered, which makes it difficult to find the 
> correct test, and also makes it difficult to introduce further improvements.
> Currently CLI tests are in the following classes:
>  * tests.cli.test_cli.TestCLI
>  * tests.core.TestCli
> I would like to take action to group the tests according to the base command, 
> which creates the following classes:
>  * tests.cli.test_cli.TestConnections
>  * tests.cli.test_cli.TestDags
>  * tests.cli.test_cli.TestDb
>  * tests.cli.test_cli.TestFlower
>  * tests.cli.test_cli.TestKerberos
>  * tests.cli.test_cli.TestPools
>  * tests.cli.test_cli.TestRoles
>  * tests.cli.test_cli.TestRotateFernetKey
>  * tests.cli.test_cli.TestScheduler
>  * tests.cli.test_cli.TestServeLogs
>  * tests.cli.test_cli.TestSyncPerm
>  * tests.cli.test_cli.TestTasks
>  * tests.cli.test_cli.TestUsers
>  * tests.cli.test_cli.TestVariables
>  * tests.cli.test_cli.TestVersion
>  * tests.cli.test_cli.TestWebserver
>  * tests.cli.test_cli.TestWorker
> The change will make it easier to make subsequent changes for this component: 
> In particular, split the component cli.py file into separate modules, and 
> then introduce lazy module loading. Currently, all modules are loaded with 
> all dependencies every time. This process is time consuming.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6039) Not correctly displayed Cyrillic in the DAGs logs

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-6039:

Fix Version/s: (was: 1.10.7)

> Not correctly displayed Cyrillic in the DAGs logs
> -
>
> Key: AIRFLOW-6039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6039
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Affects Versions: 1.10.4
>Reporter: Alexey Oskin
>Assignee: Alexey Oskin
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5873) KubernetesPodOperator fixes and test

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-5873:
---
Fix Version/s: (was: 1.10.7)

Setting fix version to 2.0.0 only as this depends on AIRFLOW-4851 and that's 
not easy to backport

> KubernetesPodOperator fixes and test
> 
>
> Key: AIRFLOW-5873
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5873
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.6
>Reporter: David de L.
>Assignee: David de L.
>Priority: Major
> Fix For: 2.0.0
>
>
> There are two PR's for this task, one targeted at v1-10-test branch, and one 
> adapted to master branch.
>  * `KubernetesPodOperator` kwarg `resources` is erroneously passed to 
> `base_operator`, instead should only go to `PodGenerator`. The two have 
> different syntax. (both on `master` and `v1-10-test` branches)
>  * `resources` passed to PodGenerator [should be 
> `k8s.V1ResourceRequirements`]([https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45]),
>  which is now handled in `KubernetesPodOperator`
>  * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts 
> arbitrary values in `setattr` (not present on either branch 
> https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/contrib/operators/kubernetes_pod_operator.py#L166-L171)
>  * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and 
> refactors
>  ** e.g. move kubernetes folder one level up from `/contrib` 
> [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21]
>  ** fix `xcom_push` to `do_xcom_push` 
> [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5873) KubernetesPodOperator fixes and test

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-5873:
---
Fix Version/s: (was: 2.0.0)
   1.10.7

Sorry, hadn't seen there were two PRs and I was just looking at this from 
trying to cherry-pick the fix back.

> KubernetesPodOperator fixes and test
> 
>
> Key: AIRFLOW-5873
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5873
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.6
>Reporter: David de L.
>Assignee: David de L.
>Priority: Major
> Fix For: 1.10.7
>
>
> There are two PR's for this task, one targeted at v1-10-test branch, and one 
> adapted to master branch.
>  * `KubernetesPodOperator` kwarg `resources` is erroneously passed to 
> `base_operator`, instead should only go to `PodGenerator`. The two have 
> different syntax. (both on `master` and `v1-10-test` branches)
>  * `resources` passed to PodGenerator [should be 
> `k8s.V1ResourceRequirements`]([https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45]),
>  which is now handled in `KubernetesPodOperator`
>  * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts 
> arbitrary values in `setattr` (not present on either branch 
> https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/contrib/operators/kubernetes_pod_operator.py#L166-L171)
>  * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and 
> refactors
>  ** e.g. move kubernetes folder one level up from `/contrib` 
> [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21]
>  ** fix `xcom_push` to `do_xcom_push` 
> [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] afischer-tech commented on issue #6569: [AIRFLOW-5916] Add Password-Alias support in SqoopHook

[GitBox]

afischer-tech commented on issue #6569: [AIRFLOW-5916] Add Password-Alias 
support in SqoopHook
URL: https://github.com/apache/airflow/pull/6569#issuecomment-566499428
 
 
   How can I do that? I haven't found a guide.
   
   Em ter., 17 de dez. de 2019 às 07:34, Kamil Breguła <
   notificati...@github.com> escreveu:
   
   > Can you add tests to avoid regression?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zacharya19 commented on a change in pull request #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

[GitBox]

zacharya19 commented on a change in pull request #6489: [AIRFLOW-3959] 
[AIRFLOW-4026] Add filter by DAG tags
URL: https://github.com/apache/airflow/pull/6489#discussion_r358737522
 
 

 ##
 File path: airflow/models/dag.py
 ##
 @@ -1384,7 +1390,8 @@ def sync_to_db(self, owner=None, sync_time=None, 
session=None):
 orm_dag.default_view = self._default_view
 orm_dag.description = self.description
 orm_dag.schedule_interval = self.schedule_interval
-session.merge(orm_dag)
+orm_dag.tags = self.get_dagtags(session=session)
 
 Review comment:
   @OmerJog ARRAY type is currently working only with PostgreSQL.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zacharya19 commented on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter by DAG tags

[GitBox]

zacharya19 commented on issue #6489: [AIRFLOW-3959] [AIRFLOW-4026] Add filter 
by DAG tags
URL: https://github.com/apache/airflow/pull/6489#issuecomment-566502242
 
 
   Fixed conflicts.
   @ashb Any update on merging this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5928) hive hooks load_file short circuit

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998108#comment-16998108
 ] 

ASF subversion and git services commented on AIRFLOW-5928:
--

Commit dd65ce2d7b4ba198669c6303730d732d093d4292 in airflow's branch 
refs/heads/v1-10-test from Jiajie Zhong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dd65ce2 ]

[AIRFLOW-5928] Hive hooks load_file short circuit (#6582)

cherry picked from 2c1f8a4bf4bf233b77a4e9bdf0346d417614d3db


> hive hooks load_file short circuit
> --
>
> Key: AIRFLOW-5928
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5928
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.6
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Fix For: 1.10.7
>
>
> If function `load_file` with parameter `create` and `recreate` are `False`, 
> `hql = ''` and should not call `HiveCliHook.run_cli`
> Due to `recreate` in two `if` statement, `HiveCliHook.run_cli` only need to 
> in the last one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles

[Ash Berlin-Taylor (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998116#comment-16998116
 ] 

Ash Berlin-Taylor commented on AIRFLOW-4888:


Never really got any further with this -- 
https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 is the skeleton 
of what could become a migration framework to add this sort of migration in to 
alembic. I think I got stalled working out how to best to the models -- should 
we define classes link in the gist, or import the FAB models directly?

> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
>  Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User 
> and Viewer roles. It turns out that these roles are missing the 
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271 
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if 
> they already exist, so having some mechanism for adding permissions to roles 
> via migrations might be useful.
> As a palyground I started working on 
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure 
> if this is too complex or not. (It's also not a complete solution yet)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-3656) Airflow Web UI link to the docs should be dynamic to Airflow version

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998118#comment-16998118
 ] 

ASF subversion and git services commented on AIRFLOW-3656:
--

Commit 7ce959774c7a431d086d74b24a6f47753ab7c90e in airflow's branch 
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7ce9597 ]

[AIRFLOW-3656] Show doc link for the current installed version (#6690)

(cherry-picked from commit 15e30dd10a8c5c479a1ff90c74d0ca6d75d1baa1)


> Airflow Web UI link to the docs should be dynamic to Airflow version
> 
>
> Key: AIRFLOW-3656
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3656
> Project: Apache Airflow
>  Issue Type: Task
>  Components: ui
>Affects Versions: 1.10.1
>Reporter: jack
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.7
>
> Attachments: 0101.PNG
>
>
> Currently in the UI Docs->Documentation directs to 
> [https://airflow.incubator.apache.org/]
>  # This should be changed to [https://airflow.readthedocs.io/en/stable/]  
> because in readthedocs the user can actually select the docs version. while 
> the current one refers only to Master branch and the user can't change it nor 
> would even know it.
>  # Preferably by clicking on the Docs->Documentation it will pick up the 
> Airflow version and point directly to the user Airflow version. Meaning that 
> if the user runs Airflow 1.10.0 it will point to 
> [https://airflow.readthedocs.io/en/1.10.0/]  The airflow version is already 
> transparent in the UI (About->Version) so it shouldn't be difficult to build 
> this link.
>  
> Previous PR that was related to the doc link is (by [~kaxilnaik]):
> https://github.com/apache/airflow/pull/3050



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

codecov-io commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options 
by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566511988
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@fd124a5`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6801/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#6801   +/-   ##
   =
 Coverage  ?   84.29%   
   =
 Files ?  676   
 Lines ?38362   
 Branches  ?0   
   =
 Hits  ?32338   
 Misses? 6024   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6801/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `93.21% <100%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=footer). 
Last update 
[fd124a5...521d0c2](https://codecov.io/gh/apache/airflow/pull/6801?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6175) Executor changes queued tasks to scheduled without removing from task queue

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998123#comment-16998123
 ] 

ASF subversion and git services commented on AIRFLOW-6175:
--

Commit 7837b008a22c74beaf1ef3436010c9ce8ad6019e in airflow's branch 
refs/heads/v1-10-test from Daniel Imberman
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7837b00 ]

[AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)

(cherry-picked from commit f3bb4c31b866b15a9a87f269073f32fb5374c8e0)

Co-Authored-By: Kaxil Naik 
Co-Authored-By: Kevin Yang 


> Executor changes queued tasks to scheduled without removing from task queue
> ---
>
> Key: AIRFLOW-6175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Daniel Imberman
>Priority: Blocker
> Fix For: 1.10.7
>
>
> There is a bug caused by scheduler_jobs refactor which leads to task failure 
> and scheduler locking.
> Essentially when a there is an overflow of tasks going into the scheduler, 
> the tasks are set back to scheduled, but are not removed from the executor's 
> queued_tasks queue.
> This means that the executor will attempt to run tasks that are in the 
> scheduled state, but those tasks will fail dependency checks. Eventually the 
> queue is filled with scheduled tasks, and the scheduler can no longer run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5947) Make the json backend pluggable for DAG Serialization

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5947:

Fix Version/s: (was: 2.0.0)
   1.10.7

> Make the json backend pluggable for DAG Serialization
> -
>
> Key: AIRFLOW-5947
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5947
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core, scheduler
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.7
>
>
> Allow users the option to choose the JSON library of their choice for DAG 
> Serialization.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5947) Make the json backend pluggable for DAG Serialization

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5947:

Fix Version/s: (was: 1.10.7)
   2.0.0

> Make the json backend pluggable for DAG Serialization
> -
>
> Key: AIRFLOW-5947
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5947
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core, scheduler
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 2.0.0
>
>
> Allow users the option to choose the JSON library of their choice for DAG 
> Serialization.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6181) Add InProcessExecutor

[Kaxil Naik (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-6181.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Add InProcessExecutor
> -
>
> Key: AIRFLOW-6181
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6181
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 2.0.0
>
>
> The InProcessExecutor is meant as a debug tool and can be used from IDE.
>  It is a single process executor that queues TaskInstances
>  and executes them by running _run_raw_task method.
> Due to it's nature the executor can be sued with SQL Lite database. When used
>  with sensors the executor will change sensor mode to reschedule to do not
>  block the execution of DAG.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] OmerJog commented on issue #6795: Adjust the MASTER_URL of spark-submit in SparkSubmitHook

[GitBox]

OmerJog commented on issue #6795: Adjust the MASTER_URL of spark-submit in 
SparkSubmitHook
URL: https://github.com/apache/airflow/pull/6795#issuecomment-566530023
 
 
   @mikolaje Tests are failing:
   
   ```
   tests/contrib/hooks/test_spark_submit_hook.py:561: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 
   airflow/contrib/hooks/spark_submit_hook.py:149: in __init__
   self._connection = self._resolve_connection()
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 
   self = 
   def _resolve_connection(self):
   # Build from connection master or default to yarn if not available
   conn_data = {'master': 'yarn',
'queue': None,
'deploy_mode': None,
'spark_home': None,
'spark_binary': self._spark_binary or "spark-submit",
'namespace': None}
   
   try:
   # Master can be local, yarn, spark://HOST:PORT, 
mesos://HOST:PORT and
   # k8s://https://:
   conn = self.get_connection(self._conn_id)
   if conn.port:
   conn_data['master'] = "{}:{}".format(conn.host, conn.port)
   else:
   conn_data['master'] = conn.host
   >   if conn.uri_without_query:
   E   AttributeError: 'Connection' object has no attribute 
'uri_without_query'
   ```
   
   Also, can you please edit your commit message to:
   `[AIRFLOW-6212] Adjust the MASTER_URL of spark-submit in SparkSubmitHook`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] OmerJog commented on issue #6713: Add some r/w functionality to GCP Bigtable Hook

[GitBox]

OmerJog commented on issue #6713: Add some r/w functionality to GCP Bigtable 
Hook
URL: https://github.com/apache/airflow/pull/6713#issuecomment-566530923
 
 
   @EmilyMazo can you please fix errors
   
   ```
   airflow/gcp/hooks/bigtable.py:260:111: E501 line too long (138 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:270:111: E501 line too long (120 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:277:111: E501 line too long (137 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:288:111: E501 line too long (120 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:315:16: F821 undefined name 'ValueRegexFilter'
   airflow/gcp/hooks/bigtable.py:318:5: E303 too many blank lines (2)
   airflow/gcp/hooks/bigtable.py:318:111: E501 line too long (112 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:328:111: E501 line too long (120 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:337:111: E501 line too long (213 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:337:129: F821 undefined name 'RowFilter'
   airflow/gcp/hooks/bigtable.py:339:111: E501 line too long (111 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:356:111: E501 line too long (113 > 110 
characters)
   airflow/gcp/hooks/bigtable.py:360:111: E501 line too long (120 > 110 
characters)
   tests/gcp/hooks/test_bigtable.py:42:1: E302 expected 2 blank lines, found 1
   tests/gcp/hooks/test_bigtable.py:482:111: E501 line too long (148 > 110 
characters)
   tests/gcp/hooks/test_bigtable.py:482:141: E999 SyntaxError: positional 
argument follows keyword argument
   ```
   
   Also note that the Jira in your commit message is wrong.
   It should be `AIRFLOW-6155` not `AIRFLOW-6713`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by 
env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566532796
 
 
   I have some doubts about that one.
   
   While I understand where it comes from, it has a number of drawbacks:
   
   * the scripts should work the same way from all the entities that can run it 
(webserver, scheduler, workers) - with env variables you just need to make sure 
that they are set the same way, here you also likely have to set some 
authentication mechanism on all those entities or embed the credentials to read 
the credentials in the command itself.
   * it's security by obscurity. Once you have access to read the variables, 
and you can start/stop airflow or run airflow CLI you should be able to execute 
those commands yourself and get the credentials anyway.
   * You open up a possibility to run some dangerous operations - it's enough 
to set a variable (which user might not see) and get airflow to execute 
arbitrary command when the command is started. Of course if you can override 
someone's variables, you can probably do a lot more, but nevertheless it looks 
dangerous
   
   I wonder what other committers think about it ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6831: [AIRFLOW-6262] add on_execute_callback to operators

[GitBox]

potiuk merged pull request #6831: [AIRFLOW-6262] add on_execute_callback to 
operators
URL: https://github.com/apache/airflow/pull/6831
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6262) Add execute callback for operators

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998214#comment-16998214
 ] 

ASF GitHub Bot commented on AIRFLOW-6262:
-

potiuk commented on pull request #6831: [AIRFLOW-6262] add on_execute_callback 
to operators
URL: https://github.com/apache/airflow/pull/6831
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add execute callback for operators
> --
>
> Key: AIRFLOW-6262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6262
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> We are in the process of adding tracing events for various stages of task 
> runs. Adding an execute callback to operator will help us emit task execution 
> events to our monitoring system.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6262) Add execute callback for operators

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998216#comment-16998216
 ] 

ASF subversion and git services commented on AIRFLOW-6262:
--

Commit 3a6d5ed0bc48929c347df51bd60c819640af5eab in airflow's branch 
refs/heads/master from Qingping Hou
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3a6d5ed ]

[AIRFLOW-6262] add on_execute_callback to operators (#6831)



> Add execute callback for operators
> --
>
> Key: AIRFLOW-6262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6262
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> We are in the process of adding tracing events for various stages of task 
> runs. Adding an execute callback to operator will help us emit task execution 
> events to our monitoring system.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] maxirus commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error

[GitBox]

maxirus commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read 
time out error
URL: https://github.com/apache/airflow/pull/6643#issuecomment-566544011
 
 
   > If possible it would be nice to add some unit tests too.
   
   Will try to get to this later this week.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri response if None

[GitBox]

codecov-io commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri 
response if None
URL: https://github.com/apache/airflow/pull/6833#issuecomment-566545360
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=h1) 
Report
   > Merging 
[#6833](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/465c1f879d74972f1eb72ec06b68f282277bc3a3?src=pr&el=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6833/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff@@
   ##   master   #6833 +/-   ##
   
   - Coverage   84.59%   84.3%   -0.3% 
   
 Files 676 676 
 Lines   38337   38361 +24 
   
   - Hits32433   32342 -91 
   - Misses   59046019+115
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `91.73% <100%> (+0.21%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `73.73% <0%> (-24.99%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.8% <0%> (+0.19%)` | :arrow_up: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6833/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.59% <0%> (+0.86%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=footer). 
Last update 
[465c1f8...1859f04](https://codecov.io/gh/apache/airflow/pull/6833?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by 
env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all you points come to evaluating the core concept of sensitive data 
management in containers. IMHO that's not really a debate (or it would imply 
having it with the Kubernetes project itself). As-is, it is a best practice 
that sensitive data should only be stored in secrets. Secrets and configmaps 
are managed alongside with the application's deployment. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
is to provide the whole airflow.cfg file as a secret. That comes with a huge 
drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all you points come to evaluating the core concept of sensitive data 
management in containers. IMHO that's not really a debate (or it would imply 
having it with the Kubernetes project itself). As-is, it is a best practice 
that sensitive data should only be stored in secrets. Secrets and configmaps 
are managed alongside with the application's deployment. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
is to provide the whole airflow.cfg file as a secret. That comes with huge 
drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all your points come to evaluating the core concept of sensitive 
data management in containers. IMHO that's not really a debate (or it would 
imply having it with the Kubernetes project itself). As-is, it is a best 
practice that sensitive data should only be stored in secrets. Secrets and 
configmaps are managed alongside with the application's deployment. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
is to provide the whole airflow.cfg file as a secret. That comes with huge 
drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all your points come to evaluating the core concept of sensitive 
data management in containers. IMHO that's not really a debate (or it would 
imply having it with the Kubernetes project itself). As-is, it is a best 
practice that sensitive data should only be stored in secrets. Secrets and 
configmaps are managed alongside with the application's deployment: you would 
provide the same configmaps and secrets to all your Airflow components, the 
same way you would provide them with the same environment variables or 
airflow.cfg file. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
is to provide the whole airflow.cfg file as a secret. That comes with huge 
drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all your points come to evaluating the core concept of sensitive 
data management in containers. IMHO that's not really a debate (or it would 
imply having it with the Kubernetes project itself). As-is, it is a best 
practice that sensitive data should only be stored in secrets. Secrets and 
configmaps are managed alongside with the application's deployment: you would 
provide the same configmaps and secrets to all your Airflow components, the 
same way you would provide them with the same environment variables or 
airflow.cfg file. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
to Airflow containers in Kubernetes is to provide the whole airflow.cfg file as 
a secret. That comes with huge drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

NBardelot edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566553394
 
 
   @potiuk 
   
   I think all your points come to evaluating the core concept of sensitive 
data management in containers. IMHO that's not really a debate (or it would 
imply having it with the Kubernetes project itself). It is a best practice that 
sensitive data should only be stored in secrets. Secrets and configmaps are 
managed alongside with the application's deployment: you would provide the same 
configmaps and secrets to all your Airflow components, the same way you would 
provide them with the same environment variables or airflow.cfg file. 
   
   With the current Airflow design the only clean way to convey a configuration 
that includes sensitive data (user/password for the broker, the database etc.) 
to Airflow containers in Kubernetes is to provide the whole airflow.cfg file as 
a secret. That comes with huge drawbacks:
   
 * all non-sensitive data cannot be read/analyzed readily anymore
 * you cannot reuse secrets (if the database password is present in two 
fields, that's two locations where you need to manage it... instead of one 
secret)
 * you need a tool to generate the airflow.cfg from a template, since you 
do not want to store the version with sensitive data anywhere in clear-text
   
   It is far better to manage your database password in only one secret, and 
the script that computes the connection string as a configmap that does not 
contain the secret in clear-text.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6833: [AIRFLOW-3189] Remove schema from get_uri response if None

[GitBox]

potiuk merged pull request #6833: [AIRFLOW-3189] Remove schema from get_uri 
response if None
URL: https://github.com/apache/airflow/pull/6833
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri response if None

[GitBox]

potiuk commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri 
response if None
URL: https://github.com/apache/airflow/pull/6833#issuecomment-566563071
 
 
   Nice one ! Thanks @zuku1985 !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3189) DbAPI get_uri returns invalid uri if schema is None

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998270#comment-16998270
 ] 

ASF GitHub Bot commented on AIRFLOW-3189:
-

potiuk commented on pull request #6833: [AIRFLOW-3189] Remove schema from 
get_uri response if None
URL: https://github.com/apache/airflow/pull/6833
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DbAPI get_uri returns invalid uri if schema is None
> ---
>
> Key: AIRFLOW-3189
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3189
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.0
>Reporter: Thomas Haederle
>Assignee: Tomasz Żukowski
>Priority: Minor
>
> the current implementation of get_uri attaches the schema name to the URI 
> even if no schema was specified.
> This leads to errors in downstream functions such as returning an invalid 
> sqlalchemy engine.
> we should add a simple check, such when the schema is None, it should not be 
> appended to the URI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-3189) DbAPI get_uri returns invalid uri if schema is None

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998271#comment-16998271
 ] 

ASF subversion and git services commented on AIRFLOW-3189:
--

Commit e3bd2a2283c2448c988f621c9ce26c8786417953 in airflow's branch 
refs/heads/master from zuku1985
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e3bd2a2 ]

[AIRFLOW-3189] Remove schema from get_uri response if None (#6833)

"None" was appended to uri if schema=None. Check was added if
schema is None.

> DbAPI get_uri returns invalid uri if schema is None
> ---
>
> Key: AIRFLOW-3189
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3189
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.0
>Reporter: Thomas Haederle
>Assignee: Tomasz Żukowski
>Priority: Minor
>
> the current implementation of get_uri attaches the schema name to the URI 
> even if no schema was specified.
> This leads to errors in downstream functions such as returning an invalid 
> sqlalchemy engine.
> we should add a simple check, such when the schema is None, it should not be 
> appended to the URI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-3189) DbAPI get_uri returns invalid uri if schema is None

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998272#comment-16998272
 ] 

ASF subversion and git services commented on AIRFLOW-3189:
--

Commit a00758ebf4ac35e5f7c51411aa550039a50067a3 in airflow's branch 
refs/heads/v1-10-test from zuku1985
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a00758e ]

[AIRFLOW-3189] Remove schema from get_uri response if None (#6833)

"None" was appended to uri if schema=None. Check was added if
schema is None.

(cherry picked from commit e3bd2a2283c2448c988f621c9ce26c8786417953)


> DbAPI get_uri returns invalid uri if schema is None
> ---
>
> Key: AIRFLOW-3189
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3189
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.0
>Reporter: Thomas Haederle
>Assignee: Tomasz Żukowski
>Priority: Minor
>
> the current implementation of get_uri attaches the schema name to the URI 
> even if no schema was specified.
> This leads to errors in downstream functions such as returning an invalid 
> sqlalchemy engine.
> we should add a simple check, such when the schema is None, it should not be 
> appended to the URI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri response if None

[GitBox]

potiuk commented on issue #6833: [AIRFLOW-3189] Remove schema from get_uri 
response if None
URL: https://github.com/apache/airflow/pull/6833#issuecomment-566566366
 
 
   Also cherry-picked to 1.10.7
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3189) DbAPI get_uri returns invalid uri if schema is None

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-3189.
---
Fix Version/s: 1.10.7
   Resolution: Fixed

> DbAPI get_uri returns invalid uri if schema is None
> ---
>
> Key: AIRFLOW-3189
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3189
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.0
>Reporter: Thomas Haederle
>Assignee: Tomasz Żukowski
>Priority: Minor
> Fix For: 1.10.7
>
>
> the current implementation of get_uri attaches the schema name to the URI 
> even if no schema was specified.
> This leads to errors in downstream functions such as returning an invalid 
> sqlalchemy engine.
> we should add a simple check, such when the schema is None, it should not be 
> appended to the URI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on a change in pull request #6350: [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container

[GitBox]

ashb commented on a change in pull request #6350: [AIRFLOW-5681] Allow 
specification of a tag or hash for the git_sync init container
URL: https://github.com/apache/airflow/pull/6350#discussion_r358824555
 
 

 ##
 File path: tests/kubernetes/test_worker_configuration.py
 ##
 @@ -438,6 +438,30 @@ def test_make_pod_git_sync_credentials_secret(self):
 self.assertIn(password_env, pod.spec.init_containers[0].env,
   'The password env for git credentials did not get into 
the init container')
 
+def test_make_pod_git_sync_rev(self):
+# Tests the pod created with git_sync_credentials_secret will get into 
the init container
+self.kube_config.git_sync_rev = 'sampletag'
+self.kube_config.dags_volume_claim = None
+self.kube_config.dags_volume_host = None
+self.kube_config.dags_in_image = None
+self.kube_config.worker_fs_group = None
+self.kube_config.git_dags_folder_mount_point = 'dags'
+self.kube_config.git_sync_dest = 'repo'
+self.kube_config.git_subpath = 'path'
+
+worker_config = WorkerConfiguration(self.kube_config)
+
+pod = worker_config.make_pod("default", str(uuid.uuid4()), 
"test_pod_id", "test_dag_id",
+ "test_task_id", str(datetime.utcnow()), 
1, "bash -c 'ls /'")
+
+rev_env = k8s.V1EnvVar(
+name='GIT_SYNC_REV',
+value=self.kube_config.git_sync_credentials_secret,
 
 Review comment:
   ```suggestion
   value=self.kube_config.git_sync_rev,
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #5663: [AIRFLOW-4940] Add DynamoDB to S3 operator

[GitBox]

ashb commented on a change in pull request #5663: [AIRFLOW-4940] Add DynamoDB 
to S3 operator
URL: https://github.com/apache/airflow/pull/5663#discussion_r358828106
 
 

 ##
 File path: tests/contrib/operators/test_dynamodb_to_s3.py
 ##
 @@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from multiprocessing import SimpleQueue
 
 Review comment:
   Was there any reason why this is using something from Multiprocessing? 
Wouldn't a plain old-python list work just as well?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] OmerJog commented on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers.

[GitBox]

OmerJog commented on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should 
support init containers.
URL: https://github.com/apache/airflow/pull/6196#issuecomment-566569379
 
 
   @dimberman @potiuk release planed to 1.10.7 or 2.0 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb opened a new pull request #6836: [AIRFLOW-4940] Simplify tests of DynamoDBToS3Operator

[GitBox]

ashb opened a new pull request #6836: [AIRFLOW-4940] Simplify tests of 
DynamoDBToS3Operator
URL: https://github.com/apache/airflow/pull/6836
 
 
   There's no point using something from multiprocessing when a `[]` will
   do just fine.
   
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] https://issues.apache.org/jira/browse/AIRFLOW-4940
   
   ### Description
   
   - [x] Simplify the tests and don't use needless Py3 specific class
   
   ### Tests
   
   - [x] Existing tests pass
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #5663: [AIRFLOW-4940] Add DynamoDB to S3 operator

[GitBox]

ashb commented on a change in pull request #5663: [AIRFLOW-4940] Add DynamoDB 
to S3 operator
URL: https://github.com/apache/airflow/pull/5663#discussion_r358831769
 
 

 ##
 File path: tests/contrib/operators/test_dynamodb_to_s3.py
 ##
 @@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from multiprocessing import SimpleQueue
 
 Review comment:
   Raised as https://github.com/apache/airflow/pull/6836  (so that we can pull 
this commit in to 1.10.7)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6799: [AIRFLOW-XXX] KubernetesPodOperator: fix _extract_xcom for larger json files

[GitBox]

codecov-io edited a comment on issue #6799: [AIRFLOW-XXX] 
KubernetesPodOperator: fix _extract_xcom for larger json files
URL: https://github.com/apache/airflow/pull/6799#issuecomment-565150765
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`v1-10-test@d2a7816`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `80.38%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6799/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=tree)
   
   ```diff
   @@  Coverage Diff  @@
   ## v1-10-test#6799   +/-   ##
   =
 Coverage  ?   81.13%   
   =
 Files ?  519   
 Lines ?35347   
 Branches  ?0   
   =
 Hits  ?28678   
 Misses? 6669   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==)
 | `90% <ø> (ø)` | |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `77.94% <ø> (ø)` | |
   | 
[airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5)
 | `92.48% <ø> (ø)` | |
   | 
[airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=)
 | `88.76% <0%> (ø)` | |
   | 
[airflow/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL19faW5pdF9fLnB5)
 | `100% <100%> (ø)` | |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `94.77% <100%> (ø)` | |
   | 
[airflow/www\_rbac/app.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=)
 | `96.58% <100%> (ø)` | |
   | 
[airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==)
 | `97.36% <100%> (ø)` | |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/sentry.py](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW50cnkucHk=)
 | `89.04% <100%> (ø)` | |
   | ... and [33 
more](https://codecov.io/gh/apache/airflow/pull/6799/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=footer). 
Last update 
[d2a7816...f2b76e7](https://codecov.io/gh/apache/airflow/pull/6799?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3534) KubernetesPodOperator breaks with active log-collection for long running tasks

[jack (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998279#comment-16998279
 ] 

jack commented on AIRFLOW-3534:
---

Could be related to https://issues.apache.org/jira/browse/AIRFLOW-5571

> KubernetesPodOperator breaks with active log-collection for long running tasks
> --
>
> Key: AIRFLOW-3534
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3534
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.1
>Reporter: Christian Thiel
>Priority: Major
>  Labels: kubernetes
>
> If a KubernetesPodOperator is started with get_logs=True, the pod breaks if 
> no logs are produced after ~30 seconds due to http client timeout.
> The error occurs in two ways:
> 1. If the script doesn't write anything to stdout there are three WARNINGS of 
> the connectionpool trying to get the logs: 
> {code:python}
> // Some comments here
> 2018-12-17 15:23:15,092] {{logging_mixin.py:95}} WARNING - 2018-12-17 
> 15:23:15,092 WARNING Retrying (Retry(total=2, connect=None, read=None, 
> redirect=None, status=None)) after connection broken by 
> 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed 
> connection without response',))': 
> /k8s/clusters/local/api/v1/namespaces/my-namespace/pods/my-pod/log?container=base&follow=True&tailLines=10
> {code}
> Followed by a {code:python}http.client.RemoteDisconnected: Remote end closed 
> connection without response{code}
> originating from _monitor_pod in /contrib/kubernetes/pod_launcher.py
> Full Traceback:
> {code:python}
> Traceback (most recent call last):
>   File "/opt/conda/lib/python3.6/site-packages/airflow/models.py", line 1659, 
> in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
>  line 123, in execute
> get_logs=self.get_logs)
>   File 
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
>  line 90, in run_pod
> return self._monitor_pod(pod, get_logs)
>   File 
> "/opt/conda/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
>  line 102, in _monitor_pod
> _preload_content=False)
>   File 
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
>  line 18583, in read_namespaced_pod_log
> (data) = self.read_namespaced_pod_log_with_http_info(name, namespace, 
> **kwargs)
>   File 
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
>  line 18689, in read_namespaced_pod_log_with_http_info
> collection_formats=collection_formats)
>   File 
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content, 
> _request_timeout)
>   File 
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 155, in __call_api
> _request_timeout=_request_timeout)
>   File 
> "/opt/conda/lib/python3.6/site-packages/kubernetes/client/api_client.py", 
> line 342, in request
> headers=headers)
>   File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py", 
> line 231, in GET
> query_params=query_params)
>   File "/opt/conda/lib/python3.6/site-packages/kubernetes/client/rest.py", 
> line 205, in request
> headers=headers)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 68, 
> in request
> **urlopen_kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/request.py", line 89, 
> in request_encode_url
> return self.urlopen(method, url, **extra_kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/poolmanager.py", line 
> 322, in urlopen
> response = conn.urlopen(method, u.request_uri, **kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", 
> line 667, in urlopen
> **response_kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", 
> line 667, in urlopen
> **response_kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", 
> line 667, in urlopen
> **response_kw)
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", 
> line 638, in urlopen
> _stacktrace=sys.exc_info()[2])
>   File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 
> 398, in increment
> raise MaxRetryError(_pool, url, error or ResponseError(cause))
> urllib3.exceptions.MaxRetryError: 
> HTTPSConnectionPool(host='rancher.benteler.net', port=443): Max retries 
> exceeded with url: 
> /k8s/clusters/local/api/v1/namespaces/ou-seamless-airflow-ops/pods/sql-fmv-collector-s3-9074ac52/log?containe

[jira] [Commented] (AIRFLOW-4940) DynamoDB to S3 backup operator

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998282#comment-16998282
 ] 

ASF GitHub Bot commented on AIRFLOW-4940:
-

ashb commented on pull request #6836: [AIRFLOW-4940] Simplify tests of 
DynamoDBToS3Operator
URL: https://github.com/apache/airflow/pull/6836
 
 
   There's no point using something from multiprocessing when a `[]` will
   do just fine.
   
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] https://issues.apache.org/jira/browse/AIRFLOW-4940
   
   ### Description
   
   - [x] Simplify the tests and don't use needless Py3 specific class
   
   ### Tests
   
   - [x] Existing tests pass
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DynamoDB to S3 backup operator
> --
>
> Key: AIRFLOW-4940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4940
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: aws
>Affects Versions: 1.10.4
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.7
>
>
> Add an Airflow operator that back up DynamoDB table to S3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #6428: [AIRFLOW-5753] Add DailyLatestOnlyOperator

[GitBox]

stale[bot] closed pull request #6428: [AIRFLOW-5753] Add DailyLatestOnlyOperator
URL: https://github.com/apache/airflow/pull/6428
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5753) Add DailyLatestOnlyOperator

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998300#comment-16998300
 ] 

ASF GitHub Bot commented on AIRFLOW-5753:
-

stale[bot] commented on pull request #6428: [AIRFLOW-5753] Add 
DailyLatestOnlyOperator
URL: https://github.com/apache/airflow/pull/6428
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add DailyLatestOnlyOperator
> ---
>
> Key: AIRFLOW-5753
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5753
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: operators
>Affects Versions: 1.10.5
>Reporter: Bumsoo Kim
>Assignee: Bumsoo Kim
>Priority: Major
>
> In addition to {{LatestOnlyOperator}}, {{DailyLatestOnlyOperator}} is for 
> cases that we need to run only the latest run in the series of task run, but 
> to ensure that the task is run at least once in a day.
> This is useful when we scrape external data from third-parties on an hourly 
> basis/for multiple times in a day to keep data up-to-date. If the DAG is 
> paused for several days, we want to avoid multiple runs in a day, but still 
> want to ensure data is filled in every daily window.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] yuqian90 commented on a change in pull request #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

yuqian90 commented on a change in pull request #6633: [AIRFLOW-2279] Clear 
tasks across DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#discussion_r358843729
 
 

 ##
 File path: airflow/sensors/external_task_sensor.py
 ##
 @@ -161,3 +164,49 @@ def poke(self, context, session=None):
 
 session.commit()
 return count == len(dttm_filter)
+
+
+class ExternalTaskMarker(DummyOperator):
+"""
+Use this operator to indicate that a task on a different DAG depends on 
this task.
+When this task is cleared with "Recursive" selected, Airflow will clear 
the task on
+the other DAG and its downstream tasks recursively. Transitive 
dependencies is followed
+until the recursion_depth is reached.
+"""
 
 Review comment:
   Thanks for pointing out. I made this change as suggested. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by 
env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelo: I agree in  in the container world that is how you keep secrets. 
And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it so we 
should be careful because that feature can be used in totally unexpected ways I 
can imagine (for example) someone implementing a round-robin script that 
assigns different `sql_alchemy_conn` to different instances in order to 
distribute traffic across several databases (been there, done that). While this 
is pretty powerful, it's also super dangerous for maintenance and support of a 
product. I am pretty sceptical when in a product we say "and here we can run 
any script with arbitrary complexity to achieve whatever user wants". That does 
not seem right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what other think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman ?). Maybe that qualifies for devlist 
discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6607: [AIRFLOW-5462] Bump FAB to 2.2.0 to fix OAuth login issue

[GitBox]

feluelle commented on a change in pull request #6607: [AIRFLOW-5462] Bump FAB 
to 2.2.0 to fix OAuth login issue
URL: https://github.com/apache/airflow/pull/6607#discussion_r358843969
 
 

 ##
 File path: setup.py
 ##
 @@ -365,7 +365,7 @@ def do_setup():
 'cryptography>=0.9.3',
 'dill>=0.2.2, <0.4',
 'flask>=1.1.0, <2.0',
-'flask-appbuilder>=1.12.5, <2.0.0',
+"flask-appbuilder>=2.2.0, <2.3.0",
 
 Review comment:
   I would suggest doing it like this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree in  in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it - we should 
be careful because that feature can be used in totally unexpected ways I can 
imagine (for example) someone implementing a round-robin script that assigns 
different `sql_alchemy_conn` to different instances in order to distribute 
traffic across several databases (been there, done that). While this is pretty 
powerful, it's also super dangerous for maintenance and support of a product. I 
am pretty sceptical when in a product we say "and here we can run any script 
with arbitrary complexity to achieve whatever user wants". That does not seem 
right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what other think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman ?). Maybe that qualifies for devlist 
discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree in  in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it so we 
should be careful because that feature can be used in totally unexpected ways I 
can imagine (for example) someone implementing a round-robin script that 
assigns different `sql_alchemy_conn` to different instances in order to 
distribute traffic across several databases (been there, done that). While this 
is pretty powerful, it's also super dangerous for maintenance and support of a 
product. I am pretty sceptical when in a product we say "and here we can run 
any script with arbitrary complexity to achieve whatever user wants". That does 
not seem right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what other think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman ?). Maybe that qualifies for devlist 
discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree in  in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it - we should 
be careful because that feature can be used in totally unexpected ways. I can 
imagine (for example) someone implementing a round-robin script that assigns 
different `sql_alchemy_conn` to different instances in order to distribute 
traffic across several databases (been there, done that). While this is pretty 
powerful, it's also super dangerous for maintenance and support of a product. I 
am pretty sceptical when in a product we say "and here we can run any script 
with arbitrary complexity to achieve whatever user wants". That does not seem 
right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what other think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman ?). Maybe that qualifies for devlist 
discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree in  in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it - we should 
be careful because that feature can be used in totally unexpected ways. I can 
imagine (for example) someone implementing a round-robin script that assigns 
different `sql_alchemy_conn` to different instances in order to distribute 
traffic across several databases (been there, done that). While this is pretty 
powerful, it's also super dangerous for maintenance and support of a product. I 
am pretty sceptical when in a product we say "and here we can run any script 
with arbitrary complexity to achieve whatever user wants". That does not seem 
right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what other think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman., @mik-laj ?). Maybe that qualifies for 
devlist discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

yuqian90 commented on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if 
marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#issuecomment-566583717
 
 
   > Can we add a section with example somwhere in docs too in 
https://github.com/apache/airflow/blob/master/docs/concepts.rst or 
https://github.com/apache/airflow/tree/master/docs/howto
   
   Thanks @kaxil. @ashb  also suggested separately to add a doc page. I have 
added a documentation page for ExternalTaskSensor and ExternalTaskMarker. 
Here's a [preview] 
(https://github.com/yuqian90/airflow/blob/external_task_sensor_marker/docs/howto/operator/external.rst)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6607: [AIRFLOW-5462] Bump FAB to 2.2.0 to fix OAuth login issue

[GitBox]

feluelle commented on a change in pull request #6607: [AIRFLOW-5462] Bump FAB 
to 2.2.0 to fix OAuth login issue
URL: https://github.com/apache/airflow/pull/6607#discussion_r358843969
 
 

 ##
 File path: setup.py
 ##
 @@ -365,7 +365,7 @@ def do_setup():
 'cryptography>=0.9.3',
 'dill>=0.2.2, <0.4',
 'flask>=1.1.0, <2.0',
-'flask-appbuilder>=1.12.5, <2.0.0',
+"flask-appbuilder>=2.2.0, <2.3.0",
 
 Review comment:
   I would suggest doing it like this `"flask-appbuilder~=2.2",`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree in  in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it - we should 
be careful because that feature can be used in totally unexpected ways. I can 
imagine (for example) someone implementing a round-robin script that assigns 
different `sql_alchemy_conn` to different instances in order to distribute 
traffic across several databases (been there, done that). While this is pretty 
powerful, it's also super dangerous for maintenance and support of a product. I 
am pretty sceptical when in a product we say "and here we can run any script 
with arbitrary complexity to achieve whatever user wants". That does not seem 
right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what others think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman., @mik-laj ?). Maybe that qualifies for 
devlist discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

[GitBox]

yuqian90 edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across 
DAGs if marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#issuecomment-566583717
 
 
   > Can we add a section with example somwhere in docs too in 
https://github.com/apache/airflow/blob/master/docs/concepts.rst or 
https://github.com/apache/airflow/tree/master/docs/howto
   
   Thanks @kaxil. @ashb  also suggested separately to add a doc page. I have 
added a documentation page for ExternalTaskSensor and ExternalTaskMarker. 
Here's a 
[preview](https://github.com/yuqian90/airflow/blob/external_task_sensor_marker/docs/howto/operator/external.rst)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var

[GitBox]

potiuk edited a comment on issue #6801: [AIRFLOW-6260] Drive _cmd config 
options by env var
URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583
 
 
   @NBardelot : I agree that in the container world that is how you keep 
secrets. And I understand your need.
   
   But we seem to try to add a new generic feature that we might have to 
support in all future versions of Airflow. If people start using it - we should 
be careful because that feature can be used in totally unexpected ways. I can 
imagine (for example) someone implementing a round-robin script that assigns 
different `sql_alchemy_conn` to different instances in order to distribute 
traffic across several databases (been there, done that). While this is pretty 
powerful, it's also super dangerous for maintenance and support of a product. I 
am pretty sceptical when in a product we say "and here we can run any script 
with arbitrary complexity to achieve whatever user wants". That does not seem 
right because people will abuse it and product will have to support it 
indefinitely. 
   
   I don't like the "over-generic" approach here - seems we are killing a fly 
with a cannon gun (as we say in Polish). But again - this is more to start 
discussion than to completely abandon that approach. Maybe I am wrong and 
others have other opinion. I would love to hear what others think (@ashb , 
@kaxil, @feluelle, @Fokko, @dimberman., @mik-laj ?). Maybe that qualifies for 
devlist discussion and should be moved there?
   
   I totally agree that we should never keep many secrets in the same config 
file together with other settings, but my question is - why do you want to keep 
secrets in the .cfg file (or in your case in ENV variable) at all? Are there  
settings that you want to be kept secret that you cannot set otherwise? Which 
ones? 
   
   Currently the default way for airflow to keep the connection secrets is to 
store them in connection database (which for the security reason can be 
encrypted with symmetric Fernet Key and the secrets are not visible via the UI) 
- not to keep them in .cfg file nor in environment variables (the environment 
variables are just a convenience to allow mostly testing and debugging). This 
is also security-by-obscurity in a way (but protects from revealing the 
password in case someone gets just access to the DB). Maybe we should have 
similar approach for anything that is secret and comes from the .cfg file 
currently instead? But maybe we could use some of the "vault" apis instead-  
whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion 
about it in the devlist I recall. Maybe that's a better approach?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5557) Mongo Replica connection Hostname causing a crash

[jack (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998316#comment-16998316
 ] 

jack commented on AIRFLOW-5557:
---

Additional info of replicas hostnames in mongo 
[https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/]

[~zuku1985] care for another? same area of building the correct uri

> Mongo Replica connection Hostname causing a crash 
> --
>
> Key: AIRFLOW-5557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5557
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: models
>Affects Versions: 1.10.5
>Reporter: SANGAMESH PATIL
>Priority: Major
>
> Hi,
>  
> I tried to provide mongo hostname in replicated format as follows as hitting 
> this error.. 
>  *hello-1:27017,hello-2:27017,hello-3:27017/?replicaSet=MongoReplica*
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in 
>    args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, 
> in wrapper
>    return f(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, 
> in run
>    _run(args, dag, ti)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, 
> in _run
>    pool=args.pool,
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 73, 
> in wrapper
>    return func(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
>    result = task_copy.execute(context=context)
> File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py",
>  line 112, in execute
>    return_value = self.execute_callable()
> File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py",
>  line 117, in execute_callable
>    return self.python_callable(*self.op_args, **self.op_kwargs)
> File "/usr/local/airflow/dags/tasks/hooks/mongo.py", line 166, in __init__
>    self._hook = MongoHook(conn_id="my_db")
> File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/mongo_hook.py", 
> line 40, in __init__
>    self.connection = self.get_connection(conn_id)
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", 
> line 80, in get_connection
>    conn = random.choice(cls.get_connections(conn_id))
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", 
> line 71, in get_connections
>    conn = cls._get_connection_from_env(conn_id)
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", 
> line 66, in _get_connection_from_env
>    conn = Connection(conn_id=conn_id, uri=environment_uri)
> File "", line 4, in __init__
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 
> 428, in _initialize_instance
>    manager.dispatch.init_failure(self, args, kwargs)
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", 
> line 67, in __exit__
>    compat.reraise(exc_type, exc_value, exc_tb)
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 
> 277, in reraise
>    raise value
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 
> 425, in _initialize_instance
>    return manager.original_init(*mixed[1:], **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", 
> line 117, in __init__
>    self.parse_from_uri(uri)
> File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", 
> line 142, in parse_from_uri
>    self.port = uri_parts.port
> File "/usr/local/lib/python3.6/urllib/parse.py", line 169, in port
>    port = int(port, 10)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb merged pull request #6678: [AIRFLOW-6083] Add custom configuration to AWS lambda client

[GitBox]

ashb merged pull request #6678: [AIRFLOW-6083] Add custom configuration to AWS 
lambda client
URL: https://github.com/apache/airflow/pull/6678
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error

[GitBox]

codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix 
KubernetesJobWatcher Read time out error
URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=h1) 
Report
   > Merging 
[#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `9.09%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6643/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master#6643 +/-   ##
   =
   - Coverage   84.61%   84.31%   -0.3% 
   =
 Files 676  676 
 Lines   3833138353 +22 
   =
   - Hits3243332338 -95 
   - Misses   5898 6015+117
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `56.89% <9.09%> (-0.41%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.48% <0%> (-20.24%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==)
 | `85.33% <0%> (-2.67%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.42% <0%> (-0.58%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.08% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9fZ2NzLnB5)
 | `97.72% <0%> (+0.1%)` | :arrow_up: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=footer). 
Last update 
[8107651...981365c](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error

[GitBox]

codecov-io edited a comment on issue #6643: [AIRFLOW-6040] Fix 
KubernetesJobWatcher Read time out error
URL: https://github.com/apache/airflow/pull/6643#issuecomment-558187895
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=h1) 
Report
   > Merging 
[#6643](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `9.09%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6643/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master#6643 +/-   ##
   =
   - Coverage   84.61%   84.31%   -0.3% 
   =
 Files 676  676 
 Lines   3833138353 +22 
   =
   - Hits3243332338 -95 
   - Misses   5898 6015+117
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `56.89% <9.09%> (-0.41%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.48% <0%> (-20.24%)` | :arrow_down: |
   | 
[airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==)
 | `85.33% <0%> (-2.67%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `87.42% <0%> (-0.58%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.08% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9fZ2NzLnB5)
 | `97.72% <0%> (+0.1%)` | :arrow_up: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6643/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=footer). 
Last update 
[8107651...981365c](https://codecov.io/gh/apache/airflow/pull/6643?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-6083.

Fix Version/s: 1.10.7
   Resolution: Fixed

> AwsLambdaHook is not accepting non-default configuration
> 
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
> Fix For: 1.10.7
>
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
>  
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998317#comment-16998317
 ] 

ASF GitHub Bot commented on AIRFLOW-6083:
-

ashb commented on pull request #6678: [AIRFLOW-6083] Add custom configuration 
to AWS lambda client
URL: https://github.com/apache/airflow/pull/6678
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> AwsLambdaHook is not accepting non-default configuration
> 
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
>  
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998319#comment-16998319
 ] 

ASF subversion and git services commented on AIRFLOW-6083:
--

Commit c4c635df6906f56e01724573923e19763bb0da62 in airflow's branch 
refs/heads/master from Daumis102
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c4c635d ]

[AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. 
(#6678)

There was no way to pass custom configuration from AwsLambdaHook to lambda 
client. This was an issue when for example default lambda timeout was too short.

> AwsLambdaHook is not accepting non-default configuration
> 
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
>  
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] zachliu commented on issue #6607: [AIRFLOW-5462] Bump FAB to 2.2.0 to fix OAuth login issue

[GitBox]

zachliu commented on issue #6607: [AIRFLOW-5462] Bump FAB to 2.2.0 to fix OAuth 
login issue
URL: https://github.com/apache/airflow/pull/6607#issuecomment-566593048
 
 
   I'm still having the same error using `flask-AppBuilder 2.2.1` :sob: 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] EmilyMazo commented on issue #6713: Add some r/w functionality to GCP Bigtable Hook

[GitBox]

EmilyMazo commented on issue #6713: Add some r/w functionality to GCP Bigtable 
Hook
URL: https://github.com/apache/airflow/pull/6713#issuecomment-566593513
 
 
   Very sorry about the repeat commits to fix type annotations and other issues 
that would be picked up by testing locally- I'm having a bear of a time setting 
up the local testing environment. My apologies for the clutter! I can rebase 
and squash all of these commits when I get this cleaned up.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services