[jira] [Resolved] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region

[Kamil Bregula (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula resolved AIRFLOW-5249.

Fix Version/s: 1.10.4
   Resolution: Fixed

> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
> Fix For: 1.10.4
>
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on 
> a dataset that is not in the 'US' location my task fails with the following 
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status 
> check failed. Final error was: 404
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", 
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError:  https://www.googleapis.com/bigquery/v2/projects/PROJECT/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
>  returned "Not found: Job PROJECT:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py", 
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
>   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
> line 138, in get_first
> cur.execute(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1821, in execute
> self.job_id = self.run_query(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 849, in run_query
> return self.run_with_configuration(configuration)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without 
> specifying a location. When I run the same operator on a dataset that is in 
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments 
> and does not pass a location to the BigQueryHook, I believe this is the 
> source of the problem. 
>  
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to 
> this one, but the referenced task calls out the two operators I'm having an 
> issue with as out of scope and after commenting on that task I have not 
> received a response.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region

[Kamil Bregula (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula updated AIRFLOW-5249:
---
Component/s: gcp

> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp, operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
> Fix For: 1.10.4
>
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on 
> a dataset that is not in the 'US' location my task fails with the following 
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status 
> check failed. Final error was: 404
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", 
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError:  https://www.googleapis.com/bigquery/v2/projects/PROJECT/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
>  returned "Not found: Job PROJECT:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py", 
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
>   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
> line 138, in get_first
> cur.execute(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1821, in execute
> self.job_id = self.run_query(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 849, in run_query
> return self.run_with_configuration(configuration)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without 
> specifying a location. When I run the same operator on a dataset that is in 
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments 
> and does not pass a location to the BigQueryHook, I believe this is the 
> source of the problem. 
>  
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to 
> this one, but the referenced task calls out the two operators I'm having an 
> issue with as out of scope and after commenting on that task I have not 
> received a response.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] baolsen opened a new pull request #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI

[GitBox]

baolsen opened a new pull request #6758: [AIRFLOW-6195] Fixed TaskInstance 
attrs not correct on UI
URL: https://github.com/apache/airflow/pull/6758
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Fixed calculation of queued_dttm - was using the first/previous value from 
prior attempts, when tasks are re-queued or retried for some other reason. This 
is unexpected behavior, and although it was implemented specifically in that 
way it is not referenced elsewhere in the application so it seems this 
behaviour is also not needed. (queued_dttm is only referenced in the UI under 
Task Instance details - but even then it was not being displayed at all anyway 
due to below bug). 
   
   Fixed various fields not displaying correctly on the UI for "Task Instance 
Details". 
   Before (just displays None instead of value from DB):
   
![image](https://user-images.githubusercontent.com/26625123/70418365-e429f800-1a6b-11ea-8691-314f989a1cf1.png)
   
   After (displays value from DB):
   
![image](https://user-images.githubusercontent.com/26625123/70418299-d6747280-1a6b-11ea-8ff4-270ecbb0e8f6.png)
   
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Tested manually by looking at the UI
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6195) queued_dttm is "None" on UI, and not updated when tasks requeued

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991224#comment-16991224
 ] 

ASF GitHub Bot commented on AIRFLOW-6195:
-

baolsen commented on pull request #6758: [AIRFLOW-6195] Fixed TaskInstance 
attrs not correct on UI
URL: https://github.com/apache/airflow/pull/6758
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Fixed calculation of queued_dttm - was using the first/previous value from 
prior attempts, when tasks are re-queued or retried for some other reason. This 
is unexpected behavior, and although it was implemented specifically in that 
way it is not referenced elsewhere in the application so it seems this 
behaviour is also not needed. (queued_dttm is only referenced in the UI under 
Task Instance details - but even then it was not being displayed at all anyway 
due to below bug). 
   
   Fixed various fields not displaying correctly on the UI for "Task Instance 
Details". 
   Before (just displays None instead of value from DB):
   
![image](https://user-images.githubusercontent.com/26625123/70418365-e429f800-1a6b-11ea-8691-314f989a1cf1.png)
   
   After (displays value from DB):
   
![image](https://user-images.githubusercontent.com/26625123/70418299-d6747280-1a6b-11ea-8ff4-270ecbb0e8f6.png)
   
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Tested manually by looking at the UI
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> queued_dttm is "None" on UI, and not updated when tasks requeued
> 
>
> Key: AIRFLOW-6195
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6195
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Attachments: image-2019-12-08-14-44-56-762.png, 
> image-2019-12-08-14-45-34-266.png, image-2019-12-08-14-46-09-051.png
>
>
> When inspecting a task instance on the UI, the value for queued_dttm displays 
> as 'None' despite having a value in the DB. Also, the value for queued_dttm 
> is from when the task was first queued and not updated if it is requeued - it 
> is not clear if this is intentional behaviour or not.
> On UI:
> !image-2019-12-08-14-44-56-762.png!
> In DB:
> !image-2019-12-08-14-45-34-266.png!
> In reality, task was queued on 8 December and run shortly after.
> queued_dttm in the DB is from the very first attempt, and is not updated from 
> recent attemp

[GitHub] [airflow] codecov-io commented on issue #6755: [AIRFLOW-6199] Add GKE example with XCOM

[GitBox]

codecov-io commented on issue #6755: [AIRFLOW-6199] Add GKE example with XCOM
URL: https://github.com/apache/airflow/pull/6755#issuecomment-563120562
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=h1) 
Report
   > Merging 
[#6755](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/76862564126592ca3542b789796c861cef013f55?src=pr&el=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6755/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master#6755 +/-   ##
   =
   - Coverage   84.84%   84.55%   -0.3% 
   =
 Files 669  669 
 Lines   3785137860  +9 
   =
   - Hits3211432011-103 
   - Misses   5737 5849+112
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/gcp/example\_dags/example\_kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfa3ViZXJuZXRlc19lbmdpbmUucHk=)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.18% <0%> (-0.31%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `89.56% <0%> (-0.09%)` | :arrow_down: |
   | 
[airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/6755/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5)
 | `86.53% <0%> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=footer). 
Last update 
[7686256...5e359a5](https://codecov.io/gh/apache/airflow/pull/6755?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6755: [AIRFLOW-6199] Add GKE example with XCOM

[GitBox]

potiuk merged pull request #6755: [AIRFLOW-6199] Add GKE example with XCOM
URL: https://github.com/apache/airflow/pull/6755
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6199) Add GKE example with XCOM

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991241#comment-16991241
 ] 

ASF subversion and git services commented on AIRFLOW-6199:
--

Commit 46f3a86bc7147bf4fad327bb5e774eb7ae369c4b in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=46f3a86 ]

[AIRFLOW-6199] Add GKE example with XCOM (#6755)



> Add GKE example with XCOM
> -
>
> Key: AIRFLOW-6199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6199) Add GKE example with XCOM

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991240#comment-16991240
 ] 

ASF GitHub Bot commented on AIRFLOW-6199:
-

potiuk commented on pull request #6755: [AIRFLOW-6199] Add GKE example with XCOM
URL: https://github.com/apache/airflow/pull/6755
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add GKE example with XCOM
> -
>
> Key: AIRFLOW-6199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6199) Add GKE example with XCOM

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6199.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Add GKE example with XCOM
> -
>
> Key: AIRFLOW-6199
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6199
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6749: [AIRFLOW-6193] Do not use asserts in Airflow main code

[GitBox]

potiuk commented on issue #6749: [AIRFLOW-6193] Do not use asserts in Airflow 
main code
URL: https://github.com/apache/airflow/pull/6749#issuecomment-563127891
 
 
   We are still waiting for the voting to finish, but I think the result is 
clear. I hope we can merge it right after voting finishes (6 hours from now). 
Would love your reviews :).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6196) Use new syntax for NamedTuple

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991247#comment-16991247
 ] 

ASF GitHub Bot commented on AIRFLOW-6196:
-

potiuk commented on pull request #6751: [AIRFLOW-6196] Use new syntax for 
NamedTuple
URL: https://github.com/apache/airflow/pull/6751
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use new syntax for NamedTuple
> -
>
> Key: AIRFLOW-6196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6751: [AIRFLOW-6196] Use new syntax for NamedTuple

[GitBox]

potiuk merged pull request #6751: [AIRFLOW-6196] Use new syntax for NamedTuple
URL: https://github.com/apache/airflow/pull/6751
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6196) Use new syntax for NamedTuple

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6196.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Use new syntax for NamedTuple
> -
>
> Key: AIRFLOW-6196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6196) Use new syntax for NamedTuple

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991248#comment-16991248
 ] 

ASF subversion and git services commented on AIRFLOW-6196:
--

Commit 63d23ed1dae47ee0bbb42993ef1309d32e5b0c37 in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=63d23ed ]

[AIRFLOW-6196] Use new syntax for NamedTuple (#6751)



> Use new syntax for NamedTuple
> -
>
> Key: AIRFLOW-6196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #6754: [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema

[GitBox]

potiuk commented on a change in pull request #6754: [AIRFLOW-6185] SQLAlchemy 
Connection model schema not aligned with Alembic schema
URL: https://github.com/apache/airflow/pull/6754#discussion_r355323365
 
 

 ##
 File path: airflow/migrations/env.py
 ##
 @@ -68,8 +68,11 @@ def run_migrations_offline():
 
 """
 context.configure(
-url=settings.SQL_ALCHEMY_CONN, target_metadata=target_metadata,
-literal_binds=True, compare_type=COMPARE_TYPE)
+url=settings.SQL_ALCHEMY_CONN,
+target_metadata=target_metadata,
+literal_binds=True,
+compare_type=COMPARE_TYPE,
+render_as_batch=True)
 
 Review comment:
   Nice!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6754: [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema

[GitBox]

potiuk merged pull request #6754: [AIRFLOW-6185] SQLAlchemy Connection model 
schema not aligned with Alembic schema
URL: https://github.com/apache/airflow/pull/6754
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6754: [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema

[GitBox]

potiuk commented on a change in pull request #6754: [AIRFLOW-6185] SQLAlchemy 
Connection model schema not aligned with Alembic schema
URL: https://github.com/apache/airflow/pull/6754#discussion_r355323642
 
 

 ##
 File path: airflow/migrations/env.py
 ##
 @@ -22,6 +22,7 @@
 from alembic import context
 
 from airflow import models, settings
+from airflow.models.serialized_dag import SerializedDagModel  # noqa
 
 Review comment:
   Thanks! Did not realise that was the side effect. Will remember about that!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6185) SQLAlchemy Connection model schema not aligned with Alembic schema

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6185.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> SQLAlchemy Connection model schema not aligned with Alembic schema
> --
>
> Key: AIRFLOW-6185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6185
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.6
>Reporter: Xinbin Huang
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: connection_schema.PNG
>
>
> The SQLAlchemy model for the `Connection` table declare password as 
> `String(5000)` while Alembic migration script (i.e. `current_schema.py`) 
> declare password as `String(length=500)`, and the final table schema for 
> password is varchar(500)
>  * Alembic schema: 
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/migrations/versions/e3a246e0dc1_current_schema.py#L54]
>  * models/connections.py
>  
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/models/connection.py#L62]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6185) SQLAlchemy Connection model schema not aligned with Alembic schema

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991337#comment-16991337
 ] 

ASF GitHub Bot commented on AIRFLOW-6185:
-

potiuk commented on pull request #6754: [AIRFLOW-6185] SQLAlchemy Connection 
model schema not aligned with Alembic schema
URL: https://github.com/apache/airflow/pull/6754
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> SQLAlchemy Connection model schema not aligned with Alembic schema
> --
>
> Key: AIRFLOW-6185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6185
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.6
>Reporter: Xinbin Huang
>Priority: Blocker
> Attachments: connection_schema.PNG
>
>
> The SQLAlchemy model for the `Connection` table declare password as 
> `String(5000)` while Alembic migration script (i.e. `current_schema.py`) 
> declare password as `String(length=500)`, and the final table schema for 
> password is varchar(500)
>  * Alembic schema: 
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/migrations/versions/e3a246e0dc1_current_schema.py#L54]
>  * models/connections.py
>  
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/models/connection.py#L62]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6185) SQLAlchemy Connection model schema not aligned with Alembic schema

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991377#comment-16991377
 ] 

ASF subversion and git services commented on AIRFLOW-6185:
--

Commit e04059d88387117d85423dbfb258bbaa90220dcb in airflow's branch 
refs/heads/master from Xinbin Huang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e04059d ]

[AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic 
schema (#6754)

* [AIRFLOW-6185] SQLAlchemy Connection model schema aligned with Alembic schema


> SQLAlchemy Connection model schema not aligned with Alembic schema
> --
>
> Key: AIRFLOW-6185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6185
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.6
>Reporter: Xinbin Huang
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: connection_schema.PNG
>
>
> The SQLAlchemy model for the `Connection` table declare password as 
> `String(5000)` while Alembic migration script (i.e. `current_schema.py`) 
> declare password as `String(length=500)`, and the final table schema for 
> password is varchar(500)
>  * Alembic schema: 
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/migrations/versions/e3a246e0dc1_current_schema.py#L54]
>  * models/connections.py
>  
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/models/connection.py#L62]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6185) SQLAlchemy Connection model schema not aligned with Alembic schema

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991378#comment-16991378
 ] 

ASF subversion and git services commented on AIRFLOW-6185:
--

Commit e04059d88387117d85423dbfb258bbaa90220dcb in airflow's branch 
refs/heads/master from Xinbin Huang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e04059d ]

[AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic 
schema (#6754)

* [AIRFLOW-6185] SQLAlchemy Connection model schema aligned with Alembic schema


> SQLAlchemy Connection model schema not aligned with Alembic schema
> --
>
> Key: AIRFLOW-6185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6185
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.6
>Reporter: Xinbin Huang
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: connection_schema.PNG
>
>
> The SQLAlchemy model for the `Connection` table declare password as 
> `String(5000)` while Alembic migration script (i.e. `current_schema.py`) 
> declare password as `String(length=500)`, and the final table schema for 
> password is varchar(500)
>  * Alembic schema: 
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/migrations/versions/e3a246e0dc1_current_schema.py#L54]
>  * models/connections.py
>  
> [https://github.com/apache/airflow/blob/ce873afc22efee171a320344b5cbee657018aed0/airflow/models/connection.py#L62]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6203) BigQuery - parametrize hook tests

[Tobiasz Kedzierski (Jira)]

Tobiasz Kedzierski created AIRFLOW-6203:
---

 Summary: BigQuery - parametrize hook tests
 Key: AIRFLOW-6203
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6203
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp, hooks, tests
Affects Versions: 1.10.6
Reporter: Tobiasz Kedzierski
Assignee: Tobiasz Kedzierski






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] TobKed opened a new pull request #6759: [AIRFLOW-6203] BigQuery - parametrize hook tests

[GitBox]

TobKed opened a new pull request #6759: [AIRFLOW-6203] BigQuery - parametrize 
hook tests
URL: https://github.com/apache/airflow/pull/6759
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-6203
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6203) BigQuery - parametrize hook tests

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991384#comment-16991384
 ] 

ASF GitHub Bot commented on AIRFLOW-6203:
-

TobKed commented on pull request #6759: [AIRFLOW-6203] BigQuery - parametrize 
hook tests
URL: https://github.com/apache/airflow/pull/6759
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-6203
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BigQuery - parametrize hook tests
> -
>
> Key: AIRFLOW-6203
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6203
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, hooks, tests
>Affects Versions: 1.10.6
>Reporter: Tobiasz Kedzierski
>Assignee: Tobiasz Kedzierski
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6753: [AIRFLOW-6197] Use tabulate to display DAG Runs

[GitBox]

potiuk commented on issue #6753: [AIRFLOW-6197] Use tabulate to display DAG Runs
URL: https://github.com/apache/airflow/pull/6753#issuecomment-563135955
 
 
   NICE!
   ```
   
╒╤══╤═╤══╤═══╤══╕
   │ 42 │ scheduled__2019-12-08T00:00:00+00:00 │ running │ tutorial │ 
2019-12-08T00:00:00+00:00 │ 2019-12-09T09:09:00.142491+00:00 │
   
├┼──┼─┼──┼───┼──┤
   │  3 │ scheduled__2019-12-07T00:00:00+00:00 │ running │ tutorial │ 
2019-12-07T00:00:00+00:00 │ 2019-12-09T09:08:42.382330+00:00 │
   
╘╧══╧═╧══╧═══╧══╛
   root@d8257ffc347b:/opt/airflow# airflow dags list_runs tutorial --help
   /opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module 
is deprecated in favour of importlib; see the module's documentation for 
alternative uses
 import imp
   usage: airflow dags list_runs [-h] [--no_backfill] [--state STATE]
 [--output 
{fancy_grid,github,grid,html,jira,latex,latex_booktabs,latex_raw,mediawiki,moinmoin,orgtbl,pipe,plain,presto,psql,rst,simple,textile,tsv,youtrack}]
 dag_id
   
   positional arguments:
 dag_idThe id of the dag
   
   optional arguments:
 -h, --helpshow this help message and exit
 --no_backfill filter all the backfill dagruns given the dag id
 --state STATE Only list the dag runs corresponding to the state
 --output 
{fancy_grid,github,grid,html,jira,latex,latex_booktabs,latex_raw,mediawiki,moinmoin,orgtbl,pipe,plain,presto,psql,rst,simple,textile,tsv,youtrack}
   Output table format. The specified value is passed 
to the tabulate module (https://pypi.org/project/tabulate/). Valid values are: 
(fancy_grid|github|grid|html|jira|latex|latex_booktabs|latex_raw|mediawiki|moinmoin|orgtbl|pipe|plain|presto|psql|rst|simple|textile|tsv|youtrack)
   root@d8257ffc347b:/opt/airflow# airflow dags list_runs tutorial --output jira
   /opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module 
is deprecated in favour of importlib; see the module's documentation for 
alternative uses
 import imp
   [2019-12-09 09:09:51,923] {executor_loader.py:49} INFO - Using executor 
SequentialExecutor
   [2019-12-09 09:09:51,924] {dagbag.py:399} INFO - Filling up the DagBag from 
/opt/airflow/tests/dags
   [2019-12-09 09:09:52,144] {test_task_view_type_check.py:49} INFO - 
class_instance type: 
   [2019-12-09 09:09:52,212] {dagbag.py:268} INFO - File 
/opt/airflow/tests/dags/test_zip.zip assumed to contain no DAGs. Skipping.
   
   | 42 | scheduled__2019-12-08T00:00:00+00:00 | running | tutorial | 
2019-12-08T00:00:00+00:00 | 2019-12-09T09:09:00.142491+00:00 |
   |  3 | scheduled__2019-12-07T00:00:00+00:00 | running | tutorial | 
2019-12-07T00:00:00+00:00 | 2019-12-09T09:08:42.382330+00:00 |
   root@d8257ffc347b:/opt/airflow# airflow dags list_runs tutorial --output psql
   /opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module 
is deprecated in favour of importlib; see the module's documentation for 
alternative uses
 import imp
   [2019-12-09 09:10:03,479] {executor_loader.py:49} INFO - Using executor 
SequentialExecutor
   [2019-12-09 09:10:03,481] {dagbag.py:399} INFO - Filling up the DagBag from 
/opt/airflow/tests/dags
   [2019-12-09 09:10:03,655] {test_task_view_type_check.py:49} INFO - 
class_instance type: 
   [2019-12-09 09:10:03,705] {dagbag.py:268} INFO - File 
/opt/airflow/tests/dags/test_zip.zip assumed to contain no DAGs. Skipping.
   
   
++--+-+--+---+--+
   | 42 | scheduled__2019-12-08T00:00:00+00:00 | running | tutorial | 
2019-12-08T00:00:00+00:00 | 2019-12-09T09:09:00.142491+00:00 |
   |  3 | scheduled__2019-12-07T00:00:00+00:00 | running | tutorial | 
2019-12-07T00:00:00+00:00 | 2019-12-09T09:08:42.382330+00:00 |
   
++--+-+--+---+--+
   root@d8257ffc347b:/opt/airflow# airflow dags list_runs tutorial --output tsv
   /opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module 
is deprecated in favour of importlib; see the module's documentation for 
alternative uses
 import imp
   [2019-12-09 09:10:11,667] {executor_loader.py:49} INFO - Using executor 
SequentialExecutor
   [2019-12-09 09:10:11,668] {dagbag.py:399} INFO - Filling up the DagBag from 
/opt/airflow/tests/dags
   [2019-12-09 09:10:11,867] {test_task_view_type_check.py:49} INFO - 
class_instance type: 
   [2019-12-09 09:10:11,912] {dagbag.py:268} INFO -

[GitHub] [airflow] potiuk merged pull request #6753: [AIRFLOW-6197] Use tabulate to display DAG Runs

[GitBox]

potiuk merged pull request #6753: [AIRFLOW-6197] Use tabulate to display DAG 
Runs
URL: https://github.com/apache/airflow/pull/6753
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6752: [AIRFLOW-XXX] Minor fix to CONTRIBUTING.rst

[GitBox]

potiuk commented on issue #6752: [AIRFLOW-XXX] Minor fix to CONTRIBUTING.rst
URL: https://github.com/apache/airflow/pull/6752#issuecomment-563136354
 
 
   Thanks @baolsen !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6197) Use tabulate to display DAG Runs

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991390#comment-16991390
 ] 

ASF GitHub Bot commented on AIRFLOW-6197:
-

potiuk commented on pull request #6753: [AIRFLOW-6197] Use tabulate to display 
DAG Runs
URL: https://github.com/apache/airflow/pull/6753
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use tabulate to display DAG Runs
> 
>
> Key: AIRFLOW-6197
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6197
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6197) Use tabulate to display DAG Runs

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991391#comment-16991391
 ] 

ASF subversion and git services commented on AIRFLOW-6197:
--

Commit f6b48814abfa2dfcec0d63c04040e7ddfc5b in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f6b4881 ]

[AIRFLOW-6197] Use tabulate to display DAG Runs (#6753)



> Use tabulate to display DAG Runs
> 
>
> Key: AIRFLOW-6197
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6197
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6676: [AIRFLOW-6081] Refactor test_connection_command.py

[GitBox]

potiuk merged pull request #6676: [AIRFLOW-6081] Refactor 
test_connection_command.py
URL: https://github.com/apache/airflow/pull/6676
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6081) Refactor test_connection_command.py

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991392#comment-16991392
 ] 

ASF GitHub Bot commented on AIRFLOW-6081:
-

potiuk commented on pull request #6676: [AIRFLOW-6081] Refactor 
test_connection_command.py
URL: https://github.com/apache/airflow/pull/6676
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Refactor test_connection_command.py
> ---
>
> Key: AIRFLOW-6081
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6081
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6081) Refactor test_connection_command.py

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991393#comment-16991393
 ] 

ASF subversion and git services commented on AIRFLOW-6081:
--

Commit 9412f5907348e3e7aacca440d84f899e995849fd in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9412f59 ]

[AIRFLOW-6081] Refactor test_connection_command.py (#6676)



> Refactor test_connection_command.py
> ---
>
> Key: AIRFLOW-6081
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6081
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6691: [AIRFLOW-6129] Fix pylint errors

[GitBox]

potiuk commented on issue #6691: [AIRFLOW-6129] Fix pylint errors
URL: https://github.com/apache/airflow/pull/6691#issuecomment-563137421
 
 
   Surprisingly it stopped happening :). 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355333282
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
 
 Review comment:
   We do not need this line any more (python 3).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355336660
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   Just a thought. Should not this be part of the operators themselves? It 
makes more sense to have the operators idempotent rather than built-in the 
logic into DAGs. In this case I think It could be better to move the "decide" 
logic into the operator and have a single "Create" operator that checks if the 
DataSync already exists and performs update instead. It makes it then 
super-easy to write dags without having to add the Branch operator.
   
   We have done that for pretty much all GCP operators and we think it makes 
much more sense: See for exa ple dataproc operator - where we either create or 
use existinng cluster if already there.
   

https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/dataproc.py#L504
   
   This makes it so much better for idempotency and back-filling.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo

[GitHub] [airflow] potiuk commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI

[GitBox]

potiuk commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not 
correct on UI
URL: https://github.com/apache/airflow/pull/6758#issuecomment-563143167
 
 
   Thanks @baolsen! It looks reasonable, however I have not much experience 
with those parts of the UI so maybe others might want to take a look (@ashb 
@feluelle - I think you are a bit more familiar with it ;). 
   
   BTW. @baolsen answering your question from slack -> you cannot add reviewers 
if you are not maintainer/committer. But you can still add people in comments 
:).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip test

[GitBox]

potiuk commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip test
URL: https://github.com/apache/airflow/pull/6739#issuecomment-563146220
 
 
   @nuclearpinguin -> will you fix it in the other test as well?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6727: Add a structural dag validation example

[GitBox]

potiuk merged pull request #6727: Add a structural dag validation example
URL: https://github.com/apache/airflow/pull/6727
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6726: [AIRFLOW-6170] BranchPythonOperator does not do XCom push of returned value

[GitBox]

potiuk commented on issue #6726: [AIRFLOW-6170] BranchPythonOperator does not 
do XCom push of returned value
URL: https://github.com/apache/airflow/pull/6726#issuecomment-563148064
 
 
   One small request here @baolsen -> this is not a breaking change, so this is 
not a MUST, but it would be great to make a note about this change in behaviour 
of BranchPythonOperator in UPDATING.md. Would you please make a fixup and add 
this info?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip test

[GitBox]

nuclearpinguin commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip 
test
URL: https://github.com/apache/airflow/pull/6739#issuecomment-563150061
 
 
   @potiuk I did it on Friday but forgot to push...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] commented on issue #6427: [WIP] dont re-read on heartbeat

[GitBox]

stale[bot] commented on issue #6427: [WIP] dont re-read on heartbeat
URL: https://github.com/apache/airflow/pull/6427#issuecomment-563151348
 
 
   This issue has been automatically marked as stale because it has not had 
recent activity. It will be closed if no further activity occurs. Thank you for 
your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6157) Separate out executor protocol

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6157:
--
Summary: Separate out executor protocol  (was: Run multiple executors)

> Separate out executor protocol
> --
>
> Key: AIRFLOW-6157
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6157
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> Proof of concept to support multiple executors
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6157) Separate out executor protocol

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-6157:
--
Description: Some of the fields of executors are accessed directly in the 
main core. The protocol for executor can be extracted and used in all places 
where executors are used.   (was: Proof of concept to support multiple executors

 )

> Separate out executor protocol
> --
>
> Key: AIRFLOW-6157
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6157
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> Some of the fields of executors are accessed directly in the main core. The 
> protocol for executor can be extracted and used in all places where executors 
> are used. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6725: [AIRFLOW-6157] Support for multiple executors

[GitBox]

potiuk commented on issue #6725: [AIRFLOW-6157] Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725#issuecomment-563153400
 
 
   Multiple executors are not needed (we get rid of KNative executor) but some 
of the PR will be opened in a moment (I will retain the JIRA id)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6734: [AIRFLOW-6120] [API-21] Rename GoogleCloudBaseHook

[GitBox]

codecov-io commented on issue #6734: [AIRFLOW-6120] [API-21] Rename 
GoogleCloudBaseHook
URL: https://github.com/apache/airflow/pull/6734#issuecomment-563153522
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=h1) 
Report
   > Merging 
[#6734](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/70c2ba70d700059eb0b20cc0c652967963713c91?src=pr&el=desc)
 will **decrease** coverage by `0.55%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6734/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6734  +/-   ##
   ==
   - Coverage   84.84%   84.28%   -0.56% 
   ==
 Files 669  669  
 Lines   3785537880  +25 
   ==
   - Hits3211831929 -189 
   - Misses   5737 5951 +214
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/gcp/hooks/cloud\_build.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvY2xvdWRfYnVpbGQucHk=)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/cloud\_memorystore.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvY2xvdWRfbWVtb3J5c3RvcmUucHk=)
 | `73.83% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvY2xvdWRfc3FsLnB5)
 | `68.42% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/compute.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvY29tcHV0ZS5weQ==)
 | `86.86% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZ2NzLnB5)
 | `84.98% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/tasks.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvdGFza3MucHk=)
 | `92.17% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/video\_intelligence.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvdmlkZW9faW50ZWxsaWdlbmNlLnB5)
 | `89.47% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=)
 | `70.85% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnlfZHRzLnB5)
 | `86.2% <100%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/dlp.py](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGxwLnB5)
 | `98.69% <100%> (ø)` | :arrow_up: |
   | ... and [45 
more](https://codecov.io/gh/apache/airflow/pull/6734/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=footer). 
Last update 
[70c2ba7...cd6d44c](https://codecov.io/gh/apache/airflow/pull/6734?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk closed pull request #6725: [AIRFLOW-6157] Support for multiple executors

[GitBox]

potiuk closed pull request #6725: [AIRFLOW-6157] Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6157) Separate out executor protocol

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991438#comment-16991438
 ] 

ASF GitHub Bot commented on AIRFLOW-6157:
-

potiuk commented on pull request #6725: [AIRFLOW-6157] Support for multiple 
executors
URL: https://github.com/apache/airflow/pull/6725
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Separate out executor protocol
> --
>
> Key: AIRFLOW-6157
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6157
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> Some of the fields of executors are accessed directly in the main core. The 
> protocol for executor can be extracted and used in all places where executors 
> are used. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6734: [AIRFLOW-6120] [AIP-21] Rename GoogleCloudBaseHook

[GitBox]

potiuk merged pull request #6734: [AIRFLOW-6120] [AIP-21] Rename 
GoogleCloudBaseHook
URL: https://github.com/apache/airflow/pull/6734
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #6725: [AIRFLOW-6157] Support for multiple executors

[GitBox]

nuclearpinguin commented on issue #6725: [AIRFLOW-6157] Support for multiple 
executors
URL: https://github.com/apache/airflow/pull/6725#issuecomment-563154680
 
 
   @potiuk I just wonder about simplifying types. Can we make a static "tasks 
run" command builder so the ['airflow', 'tasks', ...] type can be abandoned? 
WDYT? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6120) Rename base service

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991442#comment-16991442
 ] 

ASF GitHub Bot commented on AIRFLOW-6120:
-

potiuk commented on pull request #6734: [AIRFLOW-6120] [AIP-21] Rename 
GoogleCloudBaseHook
URL: https://github.com/apache/airflow/pull/6734
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rename base service
> ---
>
> Key: AIRFLOW-6120
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6120
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Michał Słowikowski
>Assignee: Michał Słowikowski
>Priority: Minor
>
> This PR contains changes in these files:
> airflow/contrib/hooks/gcp_api_base_hook.py
> airflow/gcp/hooks/automl.py
> airflow/gcp/hooks/base.py
> airflow/gcp/hooks/bigquery.py
> airflow/gcp/hooks/bigquery_dts.py
> airflow/gcp/hooks/bigtable.py
> airflow/gcp/hooks/cloud_build.py
> airflow/gcp/hooks/cloud_memorystore.py
> airflow/gcp/hooks/cloud_sql.py
> airflow/gcp/hooks/cloud_storage_transfer_service.py
> airflow/gcp/hooks/compute.py
> airflow/gcp/hooks/dataflow.py
> airflow/gcp/hooks/datastore.py
> airflow/gcp/hooks/discovery_api.py
> airflow/gcp/hooks/dlp.py
> airflow/gcp/hooks/functions.py
> airflow/gcp/hooks/gcs.py
> airflow/gcp/hooks/gsheets.py
> airflow/gcp/hooks/kms.py
> airflow/gcp/hooks/kubernetes_engine.py
> airflow/gcp/hooks/mlengine.py
> airflow/gcp/hooks/spanner.py
> airflow/gcp/hooks/speech_to_text.py
> airflow/gcp/hooks/tasks.py
> airflow/gcp/hooks/text_to_speech.py
> airflow/gcp/hooks/translate.py
> airflow/gcp/hooks/video_intelligence.py
> airflow/gcp/operators/kubernetes_engine.py
> airflow/providers/google/cloud/hooks/dataproc.py
> airflow/providers/google/cloud/hooks/natural_language.py
> airflow/providers/google/cloud/hooks/pubsub.py
> airflow/providers/google/cloud/hooks/vision.py
> airflow/providers/google/marketing_platform/hooks/campaign_manager.py
> airflow/providers/google/marketing_platform/hooks/display_video.py
> airflow/providers/google/marketing_platform/hooks/search_ads.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6120) Rename base service

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991443#comment-16991443
 ] 

ASF subversion and git services commented on AIRFLOW-6120:
--

Commit ed0a14f321b9dab3554ae395c11c147258536ce8 in airflow's branch 
refs/heads/master from mislo
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ed0a14f ]

[AIRFLOW-6120] Rename GoogleCloudBaseHook (#6734)

* [AIRFLOW-6120] Rename GoogleCloudBaseHook


> Rename base service
> ---
>
> Key: AIRFLOW-6120
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6120
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Michał Słowikowski
>Assignee: Michał Słowikowski
>Priority: Minor
>
> This PR contains changes in these files:
> airflow/contrib/hooks/gcp_api_base_hook.py
> airflow/gcp/hooks/automl.py
> airflow/gcp/hooks/base.py
> airflow/gcp/hooks/bigquery.py
> airflow/gcp/hooks/bigquery_dts.py
> airflow/gcp/hooks/bigtable.py
> airflow/gcp/hooks/cloud_build.py
> airflow/gcp/hooks/cloud_memorystore.py
> airflow/gcp/hooks/cloud_sql.py
> airflow/gcp/hooks/cloud_storage_transfer_service.py
> airflow/gcp/hooks/compute.py
> airflow/gcp/hooks/dataflow.py
> airflow/gcp/hooks/datastore.py
> airflow/gcp/hooks/discovery_api.py
> airflow/gcp/hooks/dlp.py
> airflow/gcp/hooks/functions.py
> airflow/gcp/hooks/gcs.py
> airflow/gcp/hooks/gsheets.py
> airflow/gcp/hooks/kms.py
> airflow/gcp/hooks/kubernetes_engine.py
> airflow/gcp/hooks/mlengine.py
> airflow/gcp/hooks/spanner.py
> airflow/gcp/hooks/speech_to_text.py
> airflow/gcp/hooks/tasks.py
> airflow/gcp/hooks/text_to_speech.py
> airflow/gcp/hooks/translate.py
> airflow/gcp/hooks/video_intelligence.py
> airflow/gcp/operators/kubernetes_engine.py
> airflow/providers/google/cloud/hooks/dataproc.py
> airflow/providers/google/cloud/hooks/natural_language.py
> airflow/providers/google/cloud/hooks/pubsub.py
> airflow/providers/google/cloud/hooks/vision.py
> airflow/providers/google/marketing_platform/hooks/campaign_manager.py
> airflow/providers/google/marketing_platform/hooks/display_video.py
> airflow/providers/google/marketing_platform/hooks/search_ads.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6120) Rename base service

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991444#comment-16991444
 ] 

ASF subversion and git services commented on AIRFLOW-6120:
--

Commit ed0a14f321b9dab3554ae395c11c147258536ce8 in airflow's branch 
refs/heads/master from mislo
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ed0a14f ]

[AIRFLOW-6120] Rename GoogleCloudBaseHook (#6734)

* [AIRFLOW-6120] Rename GoogleCloudBaseHook


> Rename base service
> ---
>
> Key: AIRFLOW-6120
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6120
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Michał Słowikowski
>Assignee: Michał Słowikowski
>Priority: Minor
>
> This PR contains changes in these files:
> airflow/contrib/hooks/gcp_api_base_hook.py
> airflow/gcp/hooks/automl.py
> airflow/gcp/hooks/base.py
> airflow/gcp/hooks/bigquery.py
> airflow/gcp/hooks/bigquery_dts.py
> airflow/gcp/hooks/bigtable.py
> airflow/gcp/hooks/cloud_build.py
> airflow/gcp/hooks/cloud_memorystore.py
> airflow/gcp/hooks/cloud_sql.py
> airflow/gcp/hooks/cloud_storage_transfer_service.py
> airflow/gcp/hooks/compute.py
> airflow/gcp/hooks/dataflow.py
> airflow/gcp/hooks/datastore.py
> airflow/gcp/hooks/discovery_api.py
> airflow/gcp/hooks/dlp.py
> airflow/gcp/hooks/functions.py
> airflow/gcp/hooks/gcs.py
> airflow/gcp/hooks/gsheets.py
> airflow/gcp/hooks/kms.py
> airflow/gcp/hooks/kubernetes_engine.py
> airflow/gcp/hooks/mlengine.py
> airflow/gcp/hooks/spanner.py
> airflow/gcp/hooks/speech_to_text.py
> airflow/gcp/hooks/tasks.py
> airflow/gcp/hooks/text_to_speech.py
> airflow/gcp/hooks/translate.py
> airflow/gcp/hooks/video_intelligence.py
> airflow/gcp/operators/kubernetes_engine.py
> airflow/providers/google/cloud/hooks/dataproc.py
> airflow/providers/google/cloud/hooks/natural_language.py
> airflow/providers/google/cloud/hooks/pubsub.py
> airflow/providers/google/cloud/hooks/vision.py
> airflow/providers/google/marketing_platform/hooks/campaign_manager.py
> airflow/providers/google/marketing_platform/hooks/display_video.py
> airflow/providers/google/marketing_platform/hooks/search_ads.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6719: [AIRFLOW-6158] Upgrade sendgrid dependency

[GitBox]

potiuk merged pull request #6719: [AIRFLOW-6158] Upgrade sendgrid dependency
URL: https://github.com/apache/airflow/pull/6719
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6158) Upgrade sendgrid dependency

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991448#comment-16991448
 ] 

ASF GitHub Bot commented on AIRFLOW-6158:
-

potiuk commented on pull request #6719: [AIRFLOW-6158] Upgrade sendgrid 
dependency
URL: https://github.com/apache/airflow/pull/6719
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade sendgrid dependency
> ---
>
> Key: AIRFLOW-6158
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6158
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcin Szymanski
>Assignee: Marcin Szymanski
>Priority: Minor
>
> Parameter naming changed in Sendgrid client 6.0.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6158) Upgrade sendgrid dependency

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6158.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Upgrade sendgrid dependency
> ---
>
> Key: AIRFLOW-6158
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6158
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcin Szymanski
>Assignee: Marcin Szymanski
>Priority: Minor
> Fix For: 2.0.0
>
>
> Parameter naming changed in Sendgrid client 6.0.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6158) Upgrade sendgrid dependency

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991457#comment-16991457
 ] 

ASF subversion and git services commented on AIRFLOW-6158:
--

Commit 6f5c8cf30da2427170bb07eb6582dfdadf15afa0 in airflow's branch 
refs/heads/master from Marcin Szymański
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6f5c8cf ]

[AIRFLOW-6158] Upgrade sendgrid dependency (#6719)



> Upgrade sendgrid dependency
> ---
>
> Key: AIRFLOW-6158
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6158
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: dependencies
>Affects Versions: 1.10.6
>Reporter: Marcin Szymanski
>Assignee: Marcin Szymanski
>Priority: Minor
> Fix For: 2.0.0
>
>
> Parameter naming changed in Sendgrid client 6.0.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6120) Rename base service

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6120.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Rename base service
> ---
>
> Key: AIRFLOW-6120
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6120
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: gcp
>Affects Versions: 1.10.6
>Reporter: Michał Słowikowski
>Assignee: Michał Słowikowski
>Priority: Minor
> Fix For: 2.0.0
>
>
> This PR contains changes in these files:
> airflow/contrib/hooks/gcp_api_base_hook.py
> airflow/gcp/hooks/automl.py
> airflow/gcp/hooks/base.py
> airflow/gcp/hooks/bigquery.py
> airflow/gcp/hooks/bigquery_dts.py
> airflow/gcp/hooks/bigtable.py
> airflow/gcp/hooks/cloud_build.py
> airflow/gcp/hooks/cloud_memorystore.py
> airflow/gcp/hooks/cloud_sql.py
> airflow/gcp/hooks/cloud_storage_transfer_service.py
> airflow/gcp/hooks/compute.py
> airflow/gcp/hooks/dataflow.py
> airflow/gcp/hooks/datastore.py
> airflow/gcp/hooks/discovery_api.py
> airflow/gcp/hooks/dlp.py
> airflow/gcp/hooks/functions.py
> airflow/gcp/hooks/gcs.py
> airflow/gcp/hooks/gsheets.py
> airflow/gcp/hooks/kms.py
> airflow/gcp/hooks/kubernetes_engine.py
> airflow/gcp/hooks/mlengine.py
> airflow/gcp/hooks/spanner.py
> airflow/gcp/hooks/speech_to_text.py
> airflow/gcp/hooks/tasks.py
> airflow/gcp/hooks/text_to_speech.py
> airflow/gcp/hooks/translate.py
> airflow/gcp/hooks/video_intelligence.py
> airflow/gcp/operators/kubernetes_engine.py
> airflow/providers/google/cloud/hooks/dataproc.py
> airflow/providers/google/cloud/hooks/natural_language.py
> airflow/providers/google/cloud/hooks/pubsub.py
> airflow/providers/google/cloud/hooks/vision.py
> airflow/providers/google/marketing_platform/hooks/campaign_manager.py
> airflow/providers/google/marketing_platform/hooks/display_video.py
> airflow/providers/google/marketing_platform/hooks/search_ads.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6081) Refactor test_connection_command.py

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6081.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Refactor test_connection_command.py
> ---
>
> Key: AIRFLOW-6081
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6081
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6197) Use tabulate to display DAG Runs

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6197.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Use tabulate to display DAG Runs
> 
>
> Key: AIRFLOW-6197
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6197
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] baolsen commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI

[GitBox]

baolsen commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not 
correct on UI
URL: https://github.com/apache/airflow/pull/6758#issuecomment-563158798
 
 
   Thanks @potiuk . FYI my build is failing due to a flaky test (since it ran 
fine in my repo), if you could rerun for me that would be great.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk opened a new pull request #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

potiuk opened a new pull request #6760: [AIRFLOW-6157] Separate out common 
protocol for executors.
URL: https://github.com/apache/airflow/pull/6760
 
 
   Some of the fields of executors are accessed directly in the main core. The
   protocol for executor can be extracted and used in all places where executors
   are used. 
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6157) Separate out executor protocol

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991465#comment-16991465
 ] 

ASF GitHub Bot commented on AIRFLOW-6157:
-

potiuk commented on pull request #6760: [AIRFLOW-6157] Separate out common 
protocol for executors.
URL: https://github.com/apache/airflow/pull/6760
 
 
   Some of the fields of executors are accessed directly in the main core. The
   protocol for executor can be extracted and used in all places where executors
   are used. 
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Separate out executor protocol
> --
>
> Key: AIRFLOW-6157
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6157
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> Some of the fields of executors are accessed directly in the main core. The 
> protocol for executor can be extracted and used in all places where executors 
> are used. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on issue #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

potiuk commented on issue #6760: [AIRFLOW-6157] Separate out common protocol 
for executors.
URL: https://github.com/apache/airflow/pull/6760#issuecomment-563160326
 
 
   I removed MultipleExecutor but what remained from that PR is to have a 
separate Protocol (or interface) for all executors - because previously some of 
the internals of BaseExecutor implementation (queues) were used elsewhere. I 
changed it now to extract the general Protocol and use it everywhere.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements

[GitBox]

potiuk merged pull request #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy 
setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6072.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> aws_hook: Ability to set outbound proxy
> ---
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 2.0.0
>
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy 
> settings (even if these are set in system wide). 
>  
> The way to configure a proxy is to pass a botocore.config.Config object to 
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS 
> Operators, the "config" argument is not used by any operator. 
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow 
> Connection, could allow us to pass kwargs there that build the Config object 
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config" 
> parameter.
>  
> To set an outbound proxy is then as simple as adding this to your 
> extra_config:
> {code:java}
> { .. ,
>   "config":{ "proxies": {
>     "http": "http://myproxy:8080";,
>     "https": "http://myproxy:8080"; }},
>  .. }
> {code}
>  
> This needs to work both for the main boto3 clients that do task work, but 
> also during the assume_role process which also uses a boto3 client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991470#comment-16991470
 ] 

ASF GitHub Bot commented on AIRFLOW-6072:
-

potiuk commented on pull request #6686: [AIRFLOW-6072] aws_hook: Outbound http 
proxy setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> aws_hook: Ability to set outbound proxy
> ---
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy 
> settings (even if these are set in system wide). 
>  
> The way to configure a proxy is to pass a botocore.config.Config object to 
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS 
> Operators, the "config" argument is not used by any operator. 
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow 
> Connection, could allow us to pass kwargs there that build the Config object 
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config" 
> parameter.
>  
> To set an outbound proxy is then as simple as adding this to your 
> extra_config:
> {code:java}
> { .. ,
>   "config":{ "proxies": {
>     "http": "http://myproxy:8080";,
>     "https": "http://myproxy:8080"; }},
>  .. }
> {code}
>  
> This needs to work both for the main boto3 clients that do task work, but 
> also during the assume_role process which also uses a boto3 client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991471#comment-16991471
 ] 

ASF subversion and git services commented on AIRFLOW-6072:
--

Commit 4fb498f87ef89acc30f2576ebc5090ab0653159e in airflow's branch 
refs/heads/master from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4fb498f ]

[AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements 
(#6686)



> aws_hook: Ability to set outbound proxy
> ---
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 2.0.0
>
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy 
> settings (even if these are set in system wide). 
>  
> The way to configure a proxy is to pass a botocore.config.Config object to 
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS 
> Operators, the "config" argument is not used by any operator. 
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow 
> Connection, could allow us to pass kwargs there that build the Config object 
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config" 
> parameter.
>  
> To set an outbound proxy is then as simple as adding this to your 
> extra_config:
> {code:java}
> { .. ,
>   "config":{ "proxies": {
>     "http": "http://myproxy:8080";,
>     "https": "http://myproxy:8080"; }},
>  .. }
> {code}
>  
> This needs to work both for the main boto3 clients that do task work, but 
> also during the assume_role process which also uses a boto3 client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #6726: [AIRFLOW-6170] BranchPythonOperator does not do XCom push of returned value

[GitBox]

potiuk commented on a change in pull request #6726: [AIRFLOW-6170] 
BranchPythonOperator does not do XCom push of returned value
URL: https://github.com/apache/airflow/pull/6726#discussion_r355365076
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -41,6 +41,11 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### BranchPythonOperator has a return value
 
 Review comment:
   ❤️ 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] baolsen commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

baolsen commented on a change in pull request #6675: [AIRFLOW-6038] AWS 
DataSync example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355365604
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   Hi @potiuk , this is a great idea. I actually wrote it that way at first and 
then wasn't sure if it was good practice or not :) I'll gladly update it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355365932
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   👍 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI

[GitBox]

potiuk commented on issue #6758: [AIRFLOW-6195] Fixed TaskInstance attrs not 
correct on UI
URL: https://github.com/apache/airflow/pull/6758#issuecomment-563163666
 
 
   Restarted!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6725: [AIRFLOW-6157] Support for multiple executors

[GitBox]

potiuk commented on issue #6725: [AIRFLOW-6157] Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725#issuecomment-563164178
 
 
   > @potiuk I just wonder about simplifying types. Can we make a static "tasks 
run" command builder so the ['airflow', 'tasks', ...] type can be abandoned? 
WDYT?
   
   I guess this is a discussion for another PR ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip test

[GitBox]

potiuk commented on issue #6739: [AIRFLOW-6183] Fix flaky GCS hook gzip test
URL: https://github.com/apache/airflow/pull/6739#issuecomment-563164416
 
 
   > @potiuk I did it on Friday but forgot to push...
   
   :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] baolsen commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

baolsen commented on a change in pull request #6675: [AIRFLOW-6038] AWS 
DataSync example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355369948
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   I just remembered, previously when there was one operator the build was 
failing because there were too many parameters to it. I think the limit was 
about 10, but I was using +-12. 
   Do you think this a big issue, or can just disable that check?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6725: [AIRFLOW-6157] Support for multiple executors

[GitBox]

mik-laj commented on issue #6725: [AIRFLOW-6157] Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725#issuecomment-563168109
 
 
   @nuclearpinguin It looks like one of the tasks that we can do for our second 
client.  I think, however, that the executor should not be based on sending 
commands, but objects that describe the launch, and only the worker should 
build the appropriate command.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] elukey commented on issue #5659: [AIRFLOW-5033] Switched to snakebite-py3 [DO NOT MERGE]

[GitBox]

elukey commented on issue #5659: [AIRFLOW-5033] Switched to snakebite-py3 [DO 
NOT MERGE]
URL: https://github.com/apache/airflow/pull/5659#issuecomment-563168780
 
 
   Thanks all for the comments, I'll surely try pure-sasl3! @potiuk can you add 
a comment about what I asked above? I am trying to schedule dev work on 
snakebite, but I didn't get if it blocks now Airflow 2.0 or not :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355383414
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   Just disable. It's quite OK to disable those as long as this is deliberate 
and no better alternative (say grouping parameters in an object) is found.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync example_dags added

[GitBox]

potiuk commented on a change in pull request #6675: [AIRFLOW-6038] AWS DataSync 
example_dags added
URL: https://github.com/apache/airflow/pull/6675#discussion_r355383669
 
 

 ##
 File path: 
airflow/providers/amazon/aws/example_dags/example_datasync_complex.py
 ##
 @@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using some of the the AWS DataSync operators in a 
more complex manner.
+
+- Try to get a TaskArn. If one exists, update it.
+- If no tasks exist, try to create a new DataSync Task.
+- If source and destination locations dont exist for the new task, create 
them first
+- If many tasks exist, raise an Exception
+- After getting or creating a DataSync Task, run it
+
+Specific operators used:
+* `AWSDataSyncCreateTaskOperator`
+* `AWSDataSyncGetTasksOperator`
+* `AWSDataSyncTaskOperator`
+* `AWSDataSyncUpdateTaskOperator`
+
+This DAG relies on the following environment variables:
+
+* SOURCE_LOCATION_URI - Source location URI, usually on premisis SMB or NFS
+* DESTINATION_LOCATION_URI - Destination location URI, usually S3
+* CREATE_TASK_KWARGS - Passed to boto3.create_task(**kwargs)
+* CREATE_SOURCE_LOCATION_KWARGS - Passed to boto3.create_location(**kwargs)
+* CREATE_DESTINATION_LOCATION_KWARGS - Passed to 
boto3.create_location(**kwargs)
+* UPDATE_TASK_KWARGS - Passed to boto3.update_task(**kwargs)
+"""
+
+import json
+from os import getenv
+
+from airflow import models, utils
+from airflow.exceptions import AirflowException
+from airflow.operators.python_operator import BranchPythonOperator, 
PythonOperator
+from airflow.providers.amazon.aws.operators.datasync import (
+AWSDataSyncCreateTaskOperator, AWSDataSyncGetTasksOperator, 
AWSDataSyncTaskOperator,
+AWSDataSyncUpdateTaskOperator,
+)
+
+# [START howto_operator_datasync_complex_args]
+SOURCE_LOCATION_URI = getenv(
+"SOURCE_LOCATION_URI", "smb://hostname/directory/")
+
+DESTINATION_LOCATION_URI = getenv(
+"DESTINATION_LOCATION_URI", "s3://mybucket/prefix")
+
+default_create_task_kwargs = '{"Name": "Created by Airflow"}'
+CREATE_TASK_KWARGS = json.loads(
+getenv("CREATE_TASK_KWARGS", default_create_task_kwargs)
+)
+
+default_create_source_location_kwargs = "{}"
+CREATE_SOURCE_LOCATION_KWARGS = json.loads(
+getenv("CREATE_SOURCE_LOCATION_KWARGS",
+   default_create_source_location_kwargs)
+)
+
+bucket_access_role_arn = (
+"arn:aws:iam::2223344:role/r-2223344-my-bucket-access-role"
+)
+default_destination_location_kwargs = """\
+{"S3BucketArn": "arn:aws:s3:::mybucket",
+"S3Config": {"BucketAccessRoleArn": bucket_access_role_arn}
+}"""
+CREATE_DESTINATION_LOCATION_KWARGS = json.loads(
+getenv("CREATE_DESTINATION_LOCATION_KWARGS",
+   default_destination_location_kwargs)
+)
+
+default_update_task_kwargs = '{"Name": "Updated by Airflow"}'
+UPDATE_TASK_KWARGS = json.loads(
+getenv("UPDATE_TASK_KWARGS", default_update_task_kwargs)
+)
+
+default_args = {"start_date": utils.dates.days_ago(1)}
+# [END howto_operator_datasync_complex_args]
+
+
+# [START howto_operator_datasync_complex_decide_function]
+
+
+def decide(**kwargs):
 
 Review comment:
   And pre-commits are helpful in case you have not configured them BTW.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #5659: [AIRFLOW-5033] Switched to snakebite-py3 [DO NOT MERGE]

[GitBox]

potiuk commented on issue #5659: [AIRFLOW-5033] Switched to snakebite-py3 [DO 
NOT MERGE]
URL: https://github.com/apache/airflow/pull/5659#issuecomment-563182966
 
 
   Sorry for the delay @elukey ! Been swamped recently :). I think as long as 
we have equivalent to what we had before in Airflow 1.10 (and py2) we are 
totally unblocked. The full-encryption might be simply a missing feature which 
can be implemented later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6612: [AIRFLOW-6018] Display task instance in table during backfilling

[GitBox]

codecov-io edited a comment on issue #6612: [AIRFLOW-6018] Display task 
instance in table during backfilling
URL: https://github.com/apache/airflow/pull/6612#issuecomment-563182830
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@9412f59`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6612/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff@@
   ## master   #6612   +/-   ##
   
 Coverage  ?   84.4%   
   
 Files ? 669   
 Lines ?   37894   
 Branches  ?   0   
   
 Hits  ?   31984   
 Misses?5910   
 Partials  ?   0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6612/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.88% <100%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=footer). 
Last update 
[9412f59...0da948b](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6612: [AIRFLOW-6018] Display task instance in table during backfilling

[GitBox]

codecov-io commented on issue #6612: [AIRFLOW-6018] Display task instance in 
table during backfilling
URL: https://github.com/apache/airflow/pull/6612#issuecomment-563182830
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@9412f59`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6612/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff@@
   ## master   #6612   +/-   ##
   
 Coverage  ?   84.4%   
   
 Files ? 669   
 Lines ?   37894   
 Branches  ?   0   
   
 Hits  ?   31984   
 Misses?5910   
 Partials  ?   0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6612/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.88% <100%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=footer). 
Last update 
[9412f59...0da948b](https://codecov.io/gh/apache/airflow/pull/6612?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6612: [AIRFLOW-6018] Display task instance in table during backfilling

[GitBox]

potiuk commented on a change in pull request #6612: [AIRFLOW-6018] Display task 
instance in table during backfilling
URL: https://github.com/apache/airflow/pull/6612#discussion_r355386072
 
 

 ##
 File path: airflow/models/taskinstance.py
 ##
 @@ -122,6 +122,9 @@ def clear_task_instances(tis,
 dr.start_date = timezone.utcnow()
 
 
+TaskInstanceKey = Tuple[str, str, datetime, int]
 
 Review comment:
   Yeah. I will do it soon/


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6612: [AIRFLOW-6018] Display task instance in table during backfilling

[GitBox]

potiuk merged pull request #6612: [AIRFLOW-6018] Display task instance in table 
during backfilling
URL: https://github.com/apache/airflow/pull/6612
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6018) Display task instance in table during backfilling

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991491#comment-16991491
 ] 

ASF subversion and git services commented on AIRFLOW-6018:
--

Commit da088b3b9f7e54397c4e4242f1933e20151ae47b in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=da088b3 ]

[AIRFLOW-6018] Display task instance in table during backfilling (#6612)

* [AIRFLOW-6018] Display task instance in table during backfilling


> Display task instance in table during backfilling
> -
>
> Key: AIRFLOW-6018
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6018
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6018) Display task instance in table during backfilling

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991490#comment-16991490
 ] 

ASF subversion and git services commented on AIRFLOW-6018:
--

Commit da088b3b9f7e54397c4e4242f1933e20151ae47b in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=da088b3 ]

[AIRFLOW-6018] Display task instance in table during backfilling (#6612)

* [AIRFLOW-6018] Display task instance in table during backfilling


> Display task instance in table during backfilling
> -
>
> Key: AIRFLOW-6018
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6018
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6018) Display task instance in table during backfilling

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991489#comment-16991489
 ] 

ASF GitHub Bot commented on AIRFLOW-6018:
-

potiuk commented on pull request #6612: [AIRFLOW-6018] Display task instance in 
table during backfilling
URL: https://github.com/apache/airflow/pull/6612
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Display task instance in table during backfilling
> -
>
> Key: AIRFLOW-6018
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6018
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6018) Display task instance in table during backfilling

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6018.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Display task instance in table during backfilling
> -
>
> Key: AIRFLOW-6018
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6018
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6204) Add GCP system tests helper

[Tomasz Urbaszek (Jira)]

Tomasz Urbaszek created AIRFLOW-6204:


 Summary: Add GCP system tests helper
 Key: AIRFLOW-6204
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6204
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp, tests
Affects Versions: 2.0.0
Reporter: Tomasz Urbaszek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] nuclearpinguin opened a new pull request #6761: [AIRFLOW-6204] Add GCP system tests helper

[GitBox]

nuclearpinguin opened a new pull request #6761: [AIRFLOW-6204] Add GCP system 
tests helper
URL: https://github.com/apache/airflow/pull/6761
 
 
   This PR adds GcpResourceHelper class that provides an cli for
   creating GCP system tests. This feature allows a dveloper
   to perform various actions of setting / tearing infrastructure.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6204) Add GCP system tests helper

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991497#comment-16991497
 ] 

ASF GitHub Bot commented on AIRFLOW-6204:
-

nuclearpinguin commented on pull request #6761: [AIRFLOW-6204] Add GCP system 
tests helper
URL: https://github.com/apache/airflow/pull/6761
 
 
   This PR adds GcpResourceHelper class that provides an cli for
   creating GCP system tests. This feature allows a dveloper
   to perform various actions of setting / tearing infrastructure.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add GCP system tests helper
> ---
>
> Key: AIRFLOW-6204
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6204
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, tests
>Affects Versions: 2.0.0
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6746: [AIRFLOW-6191] Adjust pytest verbosity in CI and local environment

[GitBox]

potiuk merged pull request #6746: [AIRFLOW-6191] Adjust pytest verbosity in CI 
and local environment
URL: https://github.com/apache/airflow/pull/6746
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6191) Adjust pytest verbosity in CI and local environment

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991508#comment-16991508
 ] 

ASF GitHub Bot commented on AIRFLOW-6191:
-

potiuk commented on pull request #6746: [AIRFLOW-6191] Adjust pytest verbosity 
in CI and local environment
URL: https://github.com/apache/airflow/pull/6746
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Adjust pytest verbosity in CI and local environment
> ---
>
> Key: AIRFLOW-6191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 2.0.0
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6191) Adjust pytest verbosity in CI and local environment

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991509#comment-16991509
 ] 

ASF subversion and git services commented on AIRFLOW-6191:
--

Commit d0879257d02a06738093045717e1c711443a94b2 in airflow's branch 
refs/heads/master from Tomek
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d087925 ]

[AIRFLOW-6191] Adjust pytest verbosity in CI and local environment (#6746)



> Adjust pytest verbosity in CI and local environment
> ---
>
> Key: AIRFLOW-6191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 2.0.0
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6191) Adjust pytest verbosity in CI and local environment

[Jarek Potiuk (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6191.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Adjust pytest verbosity in CI and local environment
> ---
>
> Key: AIRFLOW-6191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6191
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 2.0.0
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

nuclearpinguin commented on a change in pull request #6760: [AIRFLOW-6157] 
Separate out common protocol for executors.
URL: https://github.com/apache/airflow/pull/6760#discussion_r355399064
 
 

 ##
 File path: airflow/executors/base_executor.py
 ##
 @@ -42,7 +43,124 @@
 QueuedTaskInstanceType = Tuple[CommandType, int, Optional[str], 
SimpleTaskInstance]
 
 
-class BaseExecutor(LoggingMixin):
+class BaseExecutorProtocol(LoggingMixin):
+"""
+Base Protocol implemented by all executors including multiple executors.
+"""
+
+def __init__(self):
+super().__init__()
+
+def start(self):  # pragma: no cover
+"""
+Executors may need to get things started.
+"""
+raise NotImplementedError()
+
+def has_task(self, task_instance: TaskInstance) -> bool:
+"""
+Checks if a task is either queued or running in this executor.
+
+:param task_instance: TaskInstance
+:return: True if the task is known to this executor
+"""
+raise NotImplementedError()
+
+def sync(self) -> None:
+"""
+Sync will get called periodically by the heartbeat method.
+Executors should override this to perform gather statuses.
+"""
+raise NotImplementedError()
 
 Review comment:
   I just wonder if do I have to implement everything? I miss some information 
what is the purpose of the protocol. For example `sync` is usually called only 
by `heartbeat` so if I do not use it there, do I have to implement it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

nuclearpinguin commented on a change in pull request #6760: [AIRFLOW-6157] 
Separate out common protocol for executors.
URL: https://github.com/apache/airflow/pull/6760#discussion_r355399316
 
 

 ##
 File path: airflow/executors/base_executor.py
 ##
 @@ -42,7 +43,124 @@
 QueuedTaskInstanceType = Tuple[CommandType, int, Optional[str], 
SimpleTaskInstance]
 
 
-class BaseExecutor(LoggingMixin):
+class BaseExecutorProtocol(LoggingMixin):
+"""
+Base Protocol implemented by all executors including multiple executors.
 
 Review comment:
   I thought we abandoned the multiple executors? 🤔


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6740: [AIRFLOW-6181] Add InProcessExecutor

[GitBox]

potiuk commented on issue #6740: [AIRFLOW-6181] Add InProcessExecutor
URL: https://github.com/apache/airflow/pull/6740#issuecomment-563196334
 
 
   @mik-laj ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6740: [AIRFLOW-6181] Add InProcessExecutor

[GitBox]

potiuk commented on issue #6740: [AIRFLOW-6181] Add InProcessExecutor
URL: https://github.com/apache/airflow/pull/6740#issuecomment-563196282
 
 
   I'd go for it. There is low risk it will break anything and I think it is 
super useful for anyone testing DAGs. Maybe also we should announce it in 
devlist/slack the there is this new way of running the in-process executor. I 
would love to cherry-pick all those related changes (pylint & others) to 1.10.7 
as soon as possible.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

potiuk commented on a change in pull request #6760: [AIRFLOW-6157] Separate out 
common protocol for executors.
URL: https://github.com/apache/airflow/pull/6760#discussion_r355401245
 
 

 ##
 File path: airflow/executors/base_executor.py
 ##
 @@ -42,7 +43,124 @@
 QueuedTaskInstanceType = Tuple[CommandType, int, Optional[str], 
SimpleTaskInstance]
 
 
-class BaseExecutor(LoggingMixin):
+class BaseExecutorProtocol(LoggingMixin):
+"""
+Base Protocol implemented by all executors including multiple executors.
 
 Review comment:
   We did :).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6760: [AIRFLOW-6157] Separate out common protocol for executors.

[GitBox]

potiuk commented on a change in pull request #6760: [AIRFLOW-6157] Separate out 
common protocol for executors.
URL: https://github.com/apache/airflow/pull/6760#discussion_r355401450
 
 

 ##
 File path: airflow/executors/base_executor.py
 ##
 @@ -42,7 +43,124 @@
 QueuedTaskInstanceType = Tuple[CommandType, int, Optional[str], 
SimpleTaskInstance]
 
 
-class BaseExecutor(LoggingMixin):
+class BaseExecutorProtocol(LoggingMixin):
+"""
+Base Protocol implemented by all executors including multiple executors.
 
 Review comment:
   ```suggestion
   Base Protocol implemented by all executors.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services